Geewook Kim, Sanghee Park
We lifted 20 functions out of this paper's own repositories and ran 16 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| naver-ai/KCSAT-ML | canonical | 16 of 20 |
| Function | Status | Where it lives |
|---|---|---|
| classify | Ran | naver-ai/KCSAT-ML/analysis/contamination_analysis.py pointer only (licence: AGPL-3.0) · get_code("ee089fe2b6308e10") |
| classify | Ran | naver-ai/KCSAT-ML/analysis/difficulty_signal_analysis.py pointer only (licence: AGPL-3.0) · get_code("b786586bab293243") |
| collect | Ran | naver-ai/KCSAT-ML/analysis/drg.py pointer only (licence: AGPL-3.0) · get_code("1b14f78b2b91b280") |
| crop_and_concat_from_meta | Ran | naver-ai/KCSAT-ML/src/utils/image_utils.py pointer only (licence: AGPL-3.0) · get_code("4925c1d16bf123ee") |
| encode_image | Ran | naver-ai/KCSAT-ML/src/utils/image_utils.py pointer only (licence: AGPL-3.0) · get_code("0cf01235db87302e") |
| encode_image_with_type | Ran | naver-ai/KCSAT-ML/src/utils/image_utils.py pointer only (licence: AGPL-3.0) · get_code("2c222ddef9c44201") |
| evaluation_answer_async | Ran | naver-ai/KCSAT-ML/src/evaluator.py pointer only (licence: AGPL-3.0) · get_code("b7bd3ad8d9303c85") |
| fit_table | Ran | naver-ai/KCSAT-ML/analysis/difficulty_signal_analysis.py pointer only (licence: AGPL-3.0) · get_code("f44378d941906ea8") |
| flatten_with_prefix | Ran | naver-ai/KCSAT-ML/src/generator.py pointer only (licence: AGPL-3.0) · get_code("bc09dcbd5da56c1e") |
| generate_response_async | Ran | naver-ai/KCSAT-ML/src/runner.py pointer only (licence: AGPL-3.0) · get_code("7042ee578fb31b8f") |
| load_json | Ran | naver-ai/KCSAT-ML/src/evaluator.py pointer only (licence: AGPL-3.0) · get_code("88816b951ca46a4d") |
| load_json | Ran | naver-ai/KCSAT-ML/src/generator.py pointer only (licence: AGPL-3.0) · get_code("c700c0687c7ab7df") |
| parse_variant | Ran | naver-ai/KCSAT-ML/analysis/human_vs_model_error.py pointer only (licence: AGPL-3.0) · get_code("392350677d773653") |
| pretty_label | Ran | naver-ai/KCSAT-ML/analysis/human_vs_model_error.py pointer only (licence: AGPL-3.0) · get_code("48c63779ce44153d") |
| regress | Ran | naver-ai/KCSAT-ML/analysis/difficulty_signal_analysis.py pointer only (licence: AGPL-3.0) · get_code("4d266da509167653") |
| variant_score | Ran | naver-ai/KCSAT-ML/analysis/edge_case_finder.py pointer only (licence: AGPL-3.0) · get_code("439121706de49e24") |
| collect_per_item | Not yet run | naver-ai/KCSAT-ML/analysis/intelligence_emerges.py pointer only (licence: AGPL-3.0) · get_code("b3586a7111030459") |
| compute_drg | Not yet run | naver-ai/KCSAT-ML/analysis/drg.py pointer only (licence: AGPL-3.0) · get_code("f19200c013e66e91") |
| download_file | Not yet run | naver-ai/KCSAT-ML/src/generator.py pointer only (licence: AGPL-3.0) · get_code("8c4bcc32509a5203") |
| render | Not yet run | naver-ai/KCSAT-ML/analysis/intelligence_emerges.py pointer only (licence: AGPL-3.0) · get_code("c59c23dd3513d56e") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Math reasoning benchmarks have proliferated, yet most lack a per-item difficulty signal grounded in actual human performance. We introduce KCSAT-ML, a decade (2014-2025) of Korean College Scholastic Ability Test (KCSAT; Suneung) mathematics: 664 problems with a 339-item core set carrying official per-item error rates from nationwide cohorts of hundreds of thousands of examinees. We pair the benchmark with Difficulty-aligned Reasoning Gain (DRG): a score-orthogonal metric that asks whether a model's mistakes concentrate on the items humans found hard, or on items humans found easy. Together they expose, across a wide range of VLMs (and LLMs via OCR), three patterns: (i) low-budget accuracy collapses on the high-human-error tail at every model size; (ii) test-time scaling (TTS) raises token use roughly linearly with cohort error rate, while accuracy gains follow a non-monotonic curve; (iii) within a single family, TTS flips between anti-scaling on the hardest items and overthinking on easier ones -two faces of the same alignment failure. On DRG, models with near-identical accuracy can sit at near-opposite values: one model gets wrong what humans also find hard, while another solves the hardest items yet fails on items humans find easy -a contrast that aggregate accuracy hides. Our code and dataset builder will be open-sourced at https://github.com/naver-ai/KCSAT-ML.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2606.10403")
get_code_for_paper("2606.10403")
have("2606.10403")
Connect an agent — have() is free.