Ashton Anderson, Serina Chang, Jake Hofman
We lifted 7 functions out of this paper's own repositories and ran 5 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| serinachang5/interactive-eval | canonical | 5 of 7 |
| Function | Status | Where it lives |
|---|---|---|
| convert_conversation_to_str | Ran | serinachang5/interactive-eval/generate_conversations.py pointer only (licence: NONE) · get_code("4c7d56046181f764") |
| get_answers_from_checkpoints | Ran | serinachang5/interactive-eval/analyze_results.py pointer only (licence: NONE) · get_code("2cb702e21e393e8f") |
| get_checkpoints | Ran | serinachang5/interactive-eval/analyze_results.py pointer only (licence: NONE) · get_code("8689136dc08e3584") |
| get_data_from_final_storage | Ran | serinachang5/interactive-eval/analyze_results.py pointer only (licence: NONE) · get_code("8400472e53a0348d") |
| letter_only | Ran | serinachang5/interactive-eval/qa_reasoning.py pointer only (licence: NONE) · get_code("2422e0b1229b167a") |
| copy_and_paste | Not yet run | serinachang5/interactive-eval/qa_reasoning.py pointer only (licence: NONE) · get_code("d6b0770a7fd138fb") |
| extract_answer_from_response | Not yet run | serinachang5/interactive-eval/qa_reasoning.py pointer only (licence: NONE) · get_code("dbd40b5329f4e864") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
With the rapid adoption of LLM-based chatbots, there is a pressing need to evaluate what humans and LLMs can achieve together. However, standard benchmarks, such as MMLU, measure LLM capabilities in isolation (i.e., "AIalone"). Here, we design and conduct a user study to convert MMLU questions into user-AI conversations, by seeding the user with the question and having them carry out a conversation with the LLM to answer their question. We release ChatBench, a new dataset with AIalone, user-alone, and user-AI data for 396 questions and two LLMs, including 144K answers and 7,336 user-AI conversations. We find that AI-alone accuracy fails to predict user-AI accuracy, with significant differences across multiple subjects (math, physics, and moral reasoning), and we analyze the user-AI conversations to provide insight into how they diverge from AI-alone benchmarks. Finally, we show that fine-tuning a user simulator on a subset of ChatBench improves its ability to estimate user-AI accuracies, increasing correlation on heldout questions by more than 20 points, creating possibilities for scaling interactive evaluation. 1 A number is rounded to the nearest hundred. The result is 500. Which number could not be the number before it was rounded to the nearest hundred? A. 458 B. 463 C. 547 D. 559 To say that a rose petal is red is to say that it A. absorbs red B. reflects red C. emits red D. transmits red For which of these two scenarios does the main character do something clearly morally wrong? Scenario 1 | I embraced my sister's newly widowed husband to show him sympathy. Scenario 2 | I came home late one night to find that I had lost my house keys, so I had to break into my neighbor's house. A
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2504.07114")
get_code_for_paper("2504.07114")
have("2504.07114")
Connect an agent — have() is free.