Yehonatan Elisha, Bracha Shapira, Lior Rokach, Seffi Cohen, Nurit Cohen-Inger
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models (LLMs) often appear to excel on public benchmarks, but these high scores may mask an overreliance on datasetspecific surface cues rather than true language understanding. We introduce the Chameleon Benchmark Overfit Detector (C-BOD), a meta-evaluation framework designed to reveal such overfitting. C-BOD systematically rephrases benchmark inputs via a parameterized transformation that preserves semantic content and labels, enabling the detection of performance degradation indicative of superficial pattern reliance. We conduct extensive experiments across two datasets, three rephrasing models, and multiple distortion levels, evaluating 32 state-of-the-art LLMs. On the MMLU benchmark, C-BOD reveals an average performance drop of 2.75% under modest rephrasings, with over 80% of models exhibiting statistically significant differences. Notably, higherperforming models and larger LLMs tend to show greater sensitivity, suggesting a deeper dependence on benchmark-specific phrasing. Due to its dataset and model-agnostic design, C-BOD can be easily integrated into evaluation pipelines and offers a promising foundation for overfitting mitigation strategies. Our findings challenge the community to look beyond leaderboard scores and prioritize resilience
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2502.07445")
get_code_for_paper("2502.07445")
have("2502.07445")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2502.07445.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2502.07445)
Connect an agent — have() is free.