SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2503.09600 · ACL · 2025

MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented Generation System

Hanyu Wang, Bo Tang, Zhiyu Li, Feiyu Xiong, Zhaoxin Fan, Simin Niu, Jihao Zhao, Zhiyuan Ji

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Retrieval-Augmented Generation (RAG), while serving as a viable complement to large language models (LLMs), often overlooks the crucial aspect of text chunking within its pipeline. This paper initially introduces a dual-metric evaluation method, comprising Boundary Clarity and Chunk Stickiness, to enable the direct quantification of chunking quality. Leveraging this assessment method, we highlight the inherent limitations of traditional and semantic chunking in handling complex contextual nuances, thereby substantiating the necessity of integrating LLMs into chunking process. To address the inherent trade-off between computational efficiency and chunking precision in LLM-based approaches, we devise the granularity-aware Mixture-of-Chunkers (MoC) framework, which consists of a three-stage processing mechanism. Notably, our objective is to guide the chunker towards generating a structured list of chunking regular expressions, which are subsequently employed to extract chunks from the original text. Extensive experiments demonstrate that both our proposed metrics and the MoC framework effectively settle challenges of the chunking task, revealing the chunking kernel while enhancing the performance of the RAG system 1 .

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2503.09600")
get_code_for_paper("2503.09600")
have("2503.09600")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2503.09600.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2503.09600.svg)](https://syntology.ai/paper/2503.09600)

Connect an agent — have() is free.