We lifted 9 functions out of this paper's own repositories and ran 9 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| franxyao/chain-of-thought-hub | canonical | 9 of 9 |
| Function | Status | Where it lives |
|---|---|---|
| extract_ans | Ran | franxyao/chain-of-thought-hub/BBH/run_bbh_claude_instant_v1.0.py code served (permissive licence) · get_code("9ad0c8eaff610fe8") |
| extract_ans | Ran | franxyao/chain-of-thought-hub/BBH/run_bbh_claude_v1.3.py code served (permissive licence) · get_code("295b8fa768a029bc") |
| extract_ans_old | Ran | franxyao/chain-of-thought-hub/BBH/run_bbh_claude_v1.3.py code served (permissive licence) · get_code("f728a7b605f92e44") |
| format_example | Ran | franxyao/chain-of-thought-hub/MMLU/run_mmlu_llama.py code served (permissive licence) · get_code("cd763eaf1ac287e7") |
| format_subject | Ran | franxyao/chain-of-thought-hub/MMLU/run_mmlu_llama.py code served (permissive licence) · get_code("6ab745408cb8648b") |
| gen_prompt | Ran | franxyao/chain-of-thought-hub/MMLU/run_mmlu_llama.py code served (permissive licence) · get_code("3383488ea82b6c15") |
| test_answer_mmlu_ | Ran | franxyao/chain-of-thought-hub/MMLU/utils.py code served (permissive licence) · get_code("0adeff57e08a3fad") |
| test_answer_mmlu_claude | Ran | franxyao/chain-of-thought-hub/MMLU/utils.py code served (permissive licence) · get_code("db42ba0a29b6cff8") |
| test_answer_mmlu_claude_instant | Ran | franxyao/chain-of-thought-hub/MMLU/utils.py code served (permissive licence) · get_code("ff7e01003fa98d8a") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
As large language models (LLMs) are continuously being developed, their evaluation becomes increasingly important yet challenging. This work proposes Chain-of-Thought Hub, an open-source evaluation suite on the multi-step reasoning capabilities of large language models. We are interested in this setting for two reasons: (1) from the behavior of GPT and PaLM model family, we observe that complex reasoning is likely to be a key differentiator between weaker and stronger LLMs; (2) we envisage large language models to become the next-generation computational platform and foster an ecosystem of LLM-based new applications, this naturally requires the foundation models to perform complex tasks that often involve the composition of linguistic and logical operations. Our approach is to compile a suite of challenging reasoning benchmarks to track the progress of LLMs. Our current results show that: (1) model scale clearly correlates with reasoning capabilities; (2) As of May 2023, Claude-v1.3 and PaLM-2 are the only two models that are comparable with GPT-4, while open-sourced models still lag behind; (3) LLaMA-65B performs closely to code-davinci-002, indicating that with successful further development such as reinforcement learning from human feedback (RLHF), it has great potential to be close to GPT-3.5-Turbo. Our results also suggest that for the open-source efforts to catch up, the community may focus more on building better base models and exploring RLHF.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2305.17306")
get_code_for_paper("2305.17306")
have("2305.17306")
Connect an agent — have() is free.