Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions

Qiao Jin, Zhiyong Lu, Qingfang Liu, Joe Menke, Thorsten Kahnt

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 31 functions out of this paper's own repositories and ran 25 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• QingfangLiu/llm-evidence-retrieval-bias

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large language model (LLM) chatbots are increasingly used to answer clinical questions with citations to relevant clinical studies. Prior research has largely focused on citation fabrication, leaving a gap in evaluating the quality of the retrieved studies and the factors driving their selection, particularly for newer models with stronger reasoning capabilities. In this study, we evaluated three recent, general-purpose LLM chatbots: Claude Sonnet 5, Gemini 3.1 Pro, and ChatGPT GPT-5.5. We prompted the models with clinical questions adapted from 20 review questions in Issues 6 and 7 of the 2026 Cochrane Database of Systematic Reviews, simulating patient, clinician, and evidence-synthesis researcher user roles. For each review question, we queried each of the three chatbots under each of the three user roles, with four independent repetitions per chatbot-role combination, yielding 720 responses in total (3 chatbots × 3 user roles × 4 repetitions × 20 review questions). Each chatbot was asked to support its answers with primary clinical citations, which we then benchmarked against the included and excluded study sets of the corresponding Cochrane reviews. On average, a single chatbot response retrieved 39.2% ± 29.8% (mean ± SD across all 720 responses) of Cochrane included studies, while citing 5.0% ± 9.4% of Cochrane excluded studies. Recall of Cochrane included studies varied significantly by model and user role. ChatGPT achieved higher recall than Claude or Gemini (63.1% ± 29.5% vs. 37.0% ± 23.8% vs. 17.3% ± 13.1%; blocked permutation test, p = 2.0 × 10 -5 ). The researcher role yielded higher recall than the clinician or patient roles (42.8% ± 30.8% vs. 38.6% ± 28.9% vs. 36.1% ± 29.3%; p = 2.0 × 10 -5 ). Controlling for publication year, citations per year, and open-access status, sample size was the only independently significant predictor of retrieval (odds ratio 1.80 per 1-unit increase in log sample size, 95% CI 1.37-2.36, p = 2.34 × 10 -5 ). These findings suggest that while LLM chatbots can retrieve some studies identified by expert reviewers, their performance varies by model and user role, and they exhibit a bias toward clinical trials with larger sample sizes. Collected LLM responses and analysis code are available at https://github.com/QingfangLiu/llm-evidence-retrieval-bias.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2608.13786")
get_code_for_paper("2608.13786")
have("2608.13786")

Connect an agent — have() is free.