Bo Zhao, Yang Tian, Yu Zhou, Zhengpeng Shi
We lifted 9 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| Foreverskyou/ToolBench-X | canonical | 6 of 9 |
| Function | Status | Where it lives |
|---|---|---|
| build_task_key | Ran | Foreverskyou/ToolBench-X/gpt_gentask.py code served (permissive licence) · get_code("1711a1fe9bd3509f") |
| clean_code_block | Ran | Foreverskyou/ToolBench-X/gpt_genexception.py code served (permissive licence) · get_code("4f156bde3bc8bda8") |
| is_quota_error | Ran | Foreverskyou/ToolBench-X/gpt_genexception.py code served (permissive licence) · get_code("b07b817e3b5ef352") |
| parse_response_to_json | Ran | Foreverskyou/ToolBench-X/gpt_gentask.py code served (permissive licence) · get_code("53967305c83fd085") |
| safe_path_segment | Ran | Foreverskyou/ToolBench-X/gpt_genexception.py code served (permissive licence) · get_code("8f810d01950ce10c") |
| save_mock_code | Ran | Foreverskyou/ToolBench-X/gpt_gentool.py code served (permissive licence) · get_code("c1b2c526641aace7") |
| clean_ab_payload | Not yet run | Foreverskyou/ToolBench-X/export_openai_trajectory_clean.py code served (permissive licence) · get_code("f387a62d5b91f7a4") |
| clean_eval_payload | Not yet run | Foreverskyou/ToolBench-X/export_openai_trajectory_clean.py code served (permissive licence) · get_code("3c32bd1b9aebbc44") |
| clean_single_mode_payload | Not yet run | Foreverskyou/ToolBench-X/export_openai_trajectory_clean.py code served (permissive licence) · get_code("37b74baadb3b4c44") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models are increasingly deployed as agents that solve tasks by interacting with external tool environments. Although recent tool-use benchmarks increasingly cover complex task settings, they still largely assume clean, stable, and trustworthy tool environments, leaving tool-environment unreliability insufficiently examined. We introduce ToolBench-X, a benchmark for evaluating agents under recoverable reliability hazards. ToolBench-X contains executable multi-step tasks across diverse domains and sequential, parallel, and mixed workflows, each paired with deterministic tools and a canonical final answer for automatic evaluation. Starting from clean tool environments, ToolBench-X injects five structured hazard types: Specification Drift, Invocation Error, Execution Failure, Output Drift, and Cross-source Conflict. Crucially, each injected instance remains solvable through at least one valid recovery path, such as retrying, fallback, verification, or cross-checking. Experiments reveal a substantial reliability gap: agents that perform well with reliable tools often fail under recoverable hazards. Further analysis shows that failures are driven less by tool-use volume or inference budget than by limited hazard diagnosis and ineffective recovery. Targeted recovery hints recover many failed tasks, while test-time scaling yields more limited gains. These results suggest that tool-use evaluation should move beyond function-call accuracy toward task completion under unreliable tool environments. The code and data is available at https://github.com/Foreverskyou/ToolBench-X.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2606.25819")
get_code_for_paper("2606.25819")
have("2606.25819")
Connect an agent — have() is free.