We lifted 2 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| ai4life-group/med-safety-bench | canonical | 0 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| create_model_hf | Not yet run | ai4life-group/med-safety-bench/models/model_hf.py code served (permissive licence) · get_code("1f74380ce9d2c0c9") |
| llm_openai | Not yet run | ai4life-group/med-safety-bench/models/model_openai.py code served (permissive licence) · get_code("baba8de282ab624c") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
As large language models (LLMs) develop increasingly sophisticated capabilities and find applications in medical settings, it becomes important to assess their medical safety due to their far-reaching implications for personal and public health, patient safety, and human rights. However, there is little to no understanding of the notion of medical safety in the context of LLMs, let alone how to evaluate and improve it. To address this gap, we first define the notion of medical safety in LLMs based on the Principles of Medical Ethics set forth by the American Medical Association. We then leverage this understanding to introduce MedSafetyBench, the first benchmark dataset designed to measure the medical safety of LLMs. We demonstrate the utility of MedSafetyBench by using it to evaluate and improve the medical safety of LLMs. Our results show that publicly-available medical LLMs do not meet standards of medical safety and that fine-tuning them using MedSafetyBench improves their medical safety while preserving their medical performance. By introducing this new benchmark dataset, our work enables a systematic study of the state of medical safety in LLMs and motivates future work in this area, paving the way to mitigate the safety risks of LLMs in medicine. The benchmark dataset and code are available at https://github.com/AI4LIFE-GROUP/med-safety-bench.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2403.03744")
get_code_for_paper("2403.03744")
have("2403.03744")
Connect an agent — have() is free.