Shuo Wang, Maosong Sun, Kangyang Luo, Yingli Shen, Wen Lai, Xueren Zhang, Alexander Fraser
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The rapid development of multilingual large language models (LLMs) highlights the need for high-quality, diverse, and well-curated multilingual datasets. In this paper, we introduce DCAD-2000 (Data Cleaning as Anomaly Detection), a largescale multilingual corpus constructed from newly extracted Common Crawl data and existing multilingual sources. DCAD-2000 covers 2,282 languages, 46.72TB of text, and 8.63 billion documents, spanning 155 high-and medium-resource languages and 159 writing scripts. To overcome the limitations of existing data cleaning approaches, which rely on manually designed heuristic thresholds, we reframe data cleaning as an anomaly detection problem. This dynamic filtering paradigm substantially improves data quality by automatically identifying and removing noisy or anomalous content. By fine-tuning LLMs on DCAD-2000, we demonstrate notable improvements in data quality, robustness of the cleaning pipeline, and downstream performance, particularly for low-resource languages across multiple multilingual benchmarks.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2502.11546")
get_code_for_paper("2502.11546")
have("2502.11546")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2502.11546.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2502.11546)
Connect an agent — have() is free.