Junyi Li, Millicent Li, Byron Wallace, Sebastian Joseph, Chantal Shaib, Iain Marshall
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models, particularly GPT-3, are able to produce high quality summaries of general domain news articles in fewand zero-shot settings. However, it is unclear if such models are similarly capable in more specialized, high-stakes domains such as biomedicine. In this paper, we enlist domain experts (individuals with medical training) to evaluate summaries of biomedical articles generated by GPT-3, given zero supervision. We consider both single-and multi-document settings. In the former, GPT-3 is tasked with generating regular and plain-language summaries of articles describing randomized controlled trials; in the latter, we assess the degree to which GPT-3 is able to synthesize evidence reported across a collection of articles. We design an annotation scheme for evaluating model outputs, with an emphasis on assessing the factual accuracy of generated summaries. We find that while GPT-3 is able to summarize and simplify single biomedical articles faithfully, it struggles to provide accurate aggregations of findings over multiple documents. We release all data and annotations used in this work. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2305.06299")
get_code_for_paper("2305.06299")
have("2305.06299")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2305.06299.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2305.06299)
Connect an agent — have() is free.