Chi Wang, Jinsung Yoon, Tomas Pfister, Jiefeng Chen, Yue Huang, Jieyu Zhao, Linxin Song, Bhavana Dalvi Mishra
We lifted 6 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| google-research/t2j-bench | canonical | 1 of 6 |
| Function | Status | Where it lives |
|---|---|---|
| thresholds_for_profile | Ran | google-research/t2j-bench/dataset/eval/lib/metrics.py code served (permissive licence) · get_code("31e48ae1427f0d70") |
| build_generated_config | Not yet run | google-research/t2j-bench/dataset/eval/lib/generate_config.py code served (permissive licence) · get_code("e2dd092e77f6ae7d") |
| compare_arrays | Not yet run | google-research/t2j-bench/dataset/eval/lib/metrics.py code served (permissive licence) · get_code("0cb595ed07aee297") |
| compare_logits | Not yet run | google-research/t2j-bench/dataset/eval/lib/metrics.py code served (permissive licence) · get_code("1fe63ccd6e254da1") |
| load_eval_config | Not yet run | google-research/t2j-bench/dataset/eval/lib/config.py code served (permissive licence) · get_code("52b75923173d460b") |
| write_generated_config | Not yet run | google-research/t2j-bench/dataset/eval/lib/generate_config.py code served (permissive licence) · get_code("a3018bcbf6005b00") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Coding agents increasingly act as codebase-scale collaborators that can assist with codebase conversion, but this progress has exposed a critical weakness: agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks while violating the semantic contracts users actually care about. This problem is especially acute in codebase conversion, where prior evaluation is largely outcome-driven and therefore unstable: two implementations can match on a shallow outcome, such as a single forward loss, while diverging in gradients, optimizer behavior, or short-horizon training dynamics. We introduce T2J-Bench, a benchmark for codebase conversion that reformulates conversion as transfer under a fixed equivalence contract. A fixed verifier then compares source and converted codebases through three ordered stages: Spec (interface admissibility), Numeric (forward outputs, losses, gradients, and objective-specific tensors), and Behavioral (short training dynamics under fixed seeds). Across 355 blind conversion attempts, the best system reaches only 26.7-28.9% overall pass rate despite Spec pass rates up to 91.1%; a 4.7× token-budget spread yields only a 2.2× pass-rate spread; and all systems overestimate success by 66.6-97.8 points relative to the fixed evaluator. This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength. Our benchmark code is publicly available at https://github.com/google-research/t2j-bench/. Recent coding agents have moved beyond autocomplete into codebase-scale collaborators that can navigate projects, edit multiple files, run commands, and support large-scale software maintenance and modernization workflows, substantially amplifying user productivity in realistic engineering settings (Anthropic
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2605.29054")
get_code_for_paper("2605.29054")
have("2605.29054")
Connect an agent — have() is free.