SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2411.13779 · 2024

NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 12 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

RepositoryRoleRan
alex2awesome/news-interview-question-generation canonical 0 of 12
FunctionStatusWhere it lives
LLM_question_gen_prompt_loader Not yet run alex2awesome/news-interview-question-generation/LLM_question_generation.py
pointer only (licence: NONE) · get_code("0bc47936273feb09")
OUTLINE_LLM_QGen_prompt_loader Not yet run alex2awesome/news-interview-question-generation/LLM_question_generation.py
pointer only (licence: NONE) · get_code("366ee473e7d2d52e")
count_qa_pairs Not yet run alex2awesome/news-interview-question-generation/data_processing/create_task_dataset.py
pointer only (licence: NONE) · get_code("ba8480c986cf277b")
downsample_task_dataset Not yet run alex2awesome/news-interview-question-generation/data_processing/create_task_dataset.py
pointer only (licence: NONE) · get_code("e3670effbb518800")
ensure_valid_json Not yet run alex2awesome/news-interview-question-generation/chain_of_thought/split_outline.py
pointer only (licence: NONE) · get_code("ceb7f5482c32ac95")
extract_interviewer_questions Not yet run alex2awesome/news-interview-question-generation/data_processing/gpt_classify_all_questions.py
pointer only (licence: NONE) · get_code("4d18565e184c4383")
extract_interviewer_questions Not yet run alex2awesome/news-interview-question-generation/data_processing/vllm_classify_all_questions.py
pointer only (licence: NONE) · get_code("8677e6c7cef658b5")
get_classify_all_questions_taxonomy_prompt Not yet run alex2awesome/news-interview-question-generation/prompts.py
pointer only (licence: NONE) · get_code("4a69fb1616f573d8")
get_classify_taxonomy_prompt Not yet run alex2awesome/news-interview-question-generation/prompts.py
pointer only (licence: NONE) · get_code("7e16649719a6765e")
get_consistency_eval_prompt Not yet run alex2awesome/news-interview-question-generation/prompts.py
pointer only (licence: NONE) · get_code("68658280a96d53a0")
get_outline_parts Not yet run alex2awesome/news-interview-question-generation/chain_of_thought/outline_only_question_gen.py
pointer only (licence: NONE) · get_code("bd9b8e1dc1b004f1")
initialize_tokenizer Not yet run alex2awesome/news-interview-question-generation/helper_functions.py
pointer only (licence: NONE) · get_code("d4f7ffbe4d1b5250")

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in generating coherent text but often struggle with grounding language and strategic dialogue. To address this gap, we focus on journalistic interviews, a domain rich in grounding communication and abundant in data. We curate a dataset of 40,000 two-person informational interviews from NPR and CNN, and reveal that LLMs are significantly less likely than human interviewers to use acknowledgements and to pivot to higher-level questions. Realizing that a fundamental deficit exists in multi-turn planning and strategic thinking, we develop a realistic simulated environment, incorporating source personas and persuasive elements, in order to facilitate the development of agents with longer-horizon rewards. Our experiments show that while source LLMs mimic human behavior in information sharing, interviewer LLMs struggle with recognizing when questions are answered and engaging persuasively, leading to suboptimal information extraction across model size and capability. These findings underscore the need for enhancing LLMs' strategic dialogue capabilities.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2411.13779")
get_code_for_paper("2411.13779")
have("2411.13779")

Connect an agent — have() is free.