Even trillion-parameter models can’t salvage dirty data, domestic large language models are starting to redo pre-training.

Beating AI Insight News Flash. Leiphone reports that over the past six months, multiple domestic large language model (LLM) vendors have begun redoing their pre-training processes, with all issues traced to data quality. One vendor spent nearly a year training a trillion-parameter model, only to have its performance outperformed by a small model with just around 1% of the parameter count. The team ultimately found the root cause lay in the training data: web spam, duplicate corpora, and low-quality annotations had been mixed into the training set, rendering even the largest models ineffective. The case comes from an anonymous source, and the specific company has not been disclosed. Tencent Hunyuan has publicly announced a full redo of its pre-training work. Since February this year, the team has rebuilt its pre-training and reinforcement learning infrastructure. Previous media reports revealed that the old version of Hunyuan had issues including ranking manipulation data being mixed into the training set and inconsistent annotation rules. The new team redefined data standards, cleaned up existing corpora, and Hy3 listed data quality and diversity as key improvement areas. Alibaba and Baidu are also stepping up data governance efforts. Alibaba’s Qwen3 uses the Qwen2.5 series models to clean documents, and has significantly supplemented synthetic math and code data. Baidu’s Ernie 4.5 has added deduplication, low-quality filtering, data mapping, and manual review processes. Neither company has publicly stated that they had to redo work due to dirty data, but both have invested more efforts in data processing for their new-generation models.