We lifted 3 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| copy not recorded | — | 1 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| docker_image_exists | Ran | this paper's copy was not recorded; identical code first harvested from just-curieous/curie pointer only · get_code("7182f9e553560ee0") |
| execute_experiment_in_container | Not yet run | this paper's copy was not recorded; identical code first harvested from just-curieous/curie pointer only · get_code("8f28bc70f91ef05a") |
| run_docker_container | Not yet run | this paper's copy was not recorded; identical code first harvested from just-curieous/curie pointer only · get_code("d66c38b718ce68e3") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Automating AI research holds immense potential for accelerating scientific progress, yet current AI agents struggle with the complexities of rigorous, end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed to systematically evaluate AI agents on complete research experiments sourced from influential AI publications. Given a research question and incomplete starter code, EXP-Bench challenges AI agents to formulate hypotheses, design and implement experimental procedures, execute them, and analyze results. To enable the creation of such intricate and authentic tasks with high-fidelity, we design a semi-autonomous pipeline to extract and structure crucial experimental details from these research papers and their associated open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks from 51 top-tier AI research papers. Evaluations of leading LLM-based agents, such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial capabilities: while scores on individual experimental aspects such as design or implementation correctness occasionally reach 20-35%, the success rate for complete, executable experiments was a mere 0.5%. By identifying these bottlenecks and providing realistic step-by-step experiment procedures, EXP-Bench serves as a vital tool for future AI agents to improve their ability to conduct AI research experiments. EXP-Bench is open-sourced at https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2505.24785")
get_code_for_paper("2505.24785")
have("2505.24785")
Connect an agent — have() is free.