Xinyao Zhang, Manuela Castillo, Nicole Sonne Heckmann, Maurizio Sessa, Francesco Speca
We lifted 3 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| madelsu/LLM-for-pharmacoepi-study-design | canonical | 2 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| codes | Ran | madelsu/LLM-for-pharmacoepi-study-design/Revisions/06_code/reproduce_all_analyses.py pointer only (licence: NONE) · get_code("16e786641869cb9f") |
| parse_sheet | Ran | madelsu/LLM-for-pharmacoepi-study-design/Revisions/06_code/reproduce_all_analyses.py pointer only (licence: NONE) · get_code("6624dbf779156b8e") |
| find | Not yet run | madelsu/LLM-for-pharmacoepi-study-design/Revisions/06_code/reproduce_all_analyses.py pointer only (licence: NONE) · get_code("73744758737864ee") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The potential of large language models (LLMs) to automate and support pharmacoepidemiologic study design is an emerging area of interest, yet their reliability remains insufficiently characterized. General-purpose LLMs often display inaccuracies, while the comparative performance of specialized biomedical LLMs in this domain remains unknown. : This study evaluated general-purpose LLMs (GPT-4o and DeepSeek-R1) versus biomedically fine-tuned LLMs (QuantFactory/Bio-Medical-Llama-3-8B-GGUF and Irathernotsay/qwen2-1.5B-medical_qa-Finetune) using 46 protocols (2018-2024) from the HMA-EMA Catalogue and Sentinel System. Performance was assessed across relevance, logic of justification, and ontology-code agreement across multiple coding systems using Least-to-Most (LTM) and Active Prompting strategies. Results: GPT-4o and DeepSeek-R1 paired with LTM prompting achieved the highest relevance and logic of justification scores, with GPT-4o-LTM reaching a median relevance score of 4 in 8 of 9 questions for HMA-EMA protocols. Biomedical LLMs showed lower relevance overall and frequently generated insufficient justification. All LLMs demonstrated limited proficiency in ontology-code mapping, although LTM provided the most consistent improvements in reasoning stability. Off-the-shelf general-purpose LLMs currently offer superior support for pharmacoepidemiologic design compared to biomedical LLMs. Prompt strategy strongly influenced LLM performance.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2604.17988")
get_code_for_paper("2604.17988")
have("2604.17988")
Connect an agent — have() is free.