We lifted 14 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| night-chen/toolqa | canonical | 2 of 14 |
| Function | Status | Where it lives |
|---|---|---|
| clean_str | Ran | night-chen/toolqa/benchmark/ChatGPT/code/query.py code served (permissive licence) · get_code("c373c5e580ba5c6f") |
| format_step | Ran | night-chen/toolqa/benchmark/ReAct/code/agents.py code served (permissive licence) · get_code("84d28fbaf38b0a59") |
| EM | Not yet run | night-chen/toolqa/benchmark/ReAct/code/environment.py code served (permissive licence) · get_code("898fb37499d60ace") |
| dispatch_openai_requests | Not yet run | night-chen/toolqa/benchmark/ChatGPT/code/query.py code served (permissive licence) · get_code("667861b2eb74408c") |
| finish | Not yet run | night-chen/toolqa/benchmark/chameleon/run_toolqa/model.py code served (permissive licence) · get_code("ad24504e2398bb16") |
| get_codex_response | Not yet run | night-chen/toolqa/benchmark/chameleon/utilities.py code served (permissive licence) · get_code("08a9a18a15549ce9") |
| get_gpt3_response | Not yet run | night-chen/toolqa/benchmark/chameleon/utilities.py code served (permissive licence) · get_code("f7dd05e069f54905") |
| normalize_answer | Not yet run | night-chen/toolqa/benchmark/ReAct/code/environment.py code served (permissive licence) · get_code("3f453625dacae10a") |
| parse_action | Not yet run | night-chen/toolqa/benchmark/chameleon/run_toolqa/run-v2.py code served (permissive licence) · get_code("f71b690030237706") |
| parse_action | Not yet run | night-chen/toolqa/benchmark/ReAct/code/agents.py code served (permissive licence) · get_code("0cd0094dec049291") |
| parse_action | Not yet run | night-chen/toolqa/benchmark/ReAct/code/environment.py code served (permissive licence) · get_code("23aab7577a21a4cf") |
| reactLLMMock | Not yet run | night-chen/toolqa/benchmark/ReAct/code/mocks.py code served (permissive licence) · get_code("d228da106025a025") |
| reflectLLMMock | Not yet run | night-chen/toolqa/benchmark/ReAct/code/mocks.py code served (permissive licence) · get_code("48168012e82e413b") |
| safe_execute | Not yet run | night-chen/toolqa/benchmark/chameleon/utilities.py code served (permissive licence) · get_code("d9db4727cc8f0df8") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large Language Models (LLMs) have demonstrated impressive performance in various NLP tasks, but they still suffer from challenges such as hallucination and weak numerical reasoning. To overcome these challenges, external tools can be used to enhance LLMs' question-answering abilities. However, current evaluation methods do not distinguish between questions that can be answered using LLMs' internal knowledge and those that require external information through tool use. To address this issue, we introduce a new dataset called ToolQA, which is designed to faithfully evaluate LLMs' ability to use external tools for question answering. Our development of ToolQA involved a scalable, automated process for dataset curation, along with 13 specialized tools designed for interaction with external knowledge in order to answer questions. Importantly, we strive to minimize the overlap between our benchmark data and LLMs' pre-training data, enabling a more precise evaluation of LLMs' tool-use reasoning abilities. We conducted an in-depth diagnosis of existing tool-use LLMs to highlight their strengths, weaknesses, and potential improvements. Our findings set a new benchmark for evaluating LLMs and suggest new directions for future advancements. Our data and code are freely available to the broader scientific community on GitHub.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2306.13304")
get_code_for_paper("2306.13304")
have("2306.13304")
Connect an agent — have() is free.