Huaxiu Yao, Peng Xia, Yun Li, Kangyu Zhu, Hongtu Zhu, Sheng Wang
We lifted 17 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| aiming-lab/mmedpo | canonical | 5 of 15 |
| aiming-lab/MMedPO | canonical | 2 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| whitespace_tokenize | Ran | aiming-lab/mmedpo/curation/Sample_Zero-Shot_Grounding_RSNA/models/tokenization_bert.py code served (permissive licence) · get_code("cf9ffa02a42184af") |
| add_gaussian_noise | Ran | aiming-lab/MMedPO/inference/llava-med-1.5_vqa.py code served (permissive licence) · get_code("d2bfc515570f0eed") |
| create_loader_RSNA | Ran | aiming-lab/MMedPO/curation/Sample_Zero-Shot_Grounding_RSNA/dataset/dataset_RSNA.py code served (permissive licence) · get_code("c15c37b39ee4184f") |
| get_chunk | Ran | aiming-lab/mmedpo/inference/llava-med-1.5_report.py code served (permissive licence) · get_code("42a46570620cd9fa") |
| load_vocab | Ran | aiming-lab/mmedpo/curation/Sample_Zero-Shot_Grounding_RSNA/models/tokenization_bert.py code served (permissive licence) · get_code("e7fbc7a74a3457c7") |
| split_list | Ran | aiming-lab/mmedpo/inference/llava-med-1.5_report.py code served (permissive licence) · get_code("076c252c52cbb161") |
| split_to_even_chunks | Ran | aiming-lab/mmedpo/train/dpo/llava_trainer_weighted.py code served (permissive licence) · get_code("10893c4608c08075") |
| add_gaussian_noise | Not yet run | aiming-lab/mmedpo/inference/llava-med-1.5_report.py code served (permissive licence) · get_code("8f611a3a3069da33") |
| bleu | Not yet run | aiming-lab/mmedpo/eval/eval_report.py code served (permissive licence) · get_code("cd8fbb4ca9a20927") |
| generate_noised_image | Not yet run | aiming-lab/mmedpo/curation/Sample_Zero-Shot_Grounding_RSNA/inference_attention-map_score.py code served (permissive licence) · get_code("de5de3e13616d4f2") |
| get_mm_adapter_state_maybe_zero_3 | Not yet run | aiming-lab/mmedpo/train/dpo/llava_trainer_weighted.py code served (permissive licence) · get_code("bb35e3ac741bb2c9") |
| get_tokenizer | Not yet run | aiming-lab/mmedpo/curation/Sample_Zero-Shot_Grounding_RSNA/inference_attention-map_score.py code served (permissive licence) · get_code("d070ec2620683428") |
| load_jsonl | Not yet run | aiming-lab/mmedpo/eval/eval_report.py code served (permissive licence) · get_code("77678f2758141df9") |
| maybe_zero_3 | Not yet run | aiming-lab/mmedpo/train/dpo/llava_trainer_weighted.py code served (permissive licence) · get_code("735025744c1ab0cf") |
| normalize_word | Not yet run | aiming-lab/mmedpo/eval/eval_report.py code served (permissive licence) · get_code("8f1a44ab20956301") |
| score_cal | Not yet run | aiming-lab/mmedpo/curation/Sample_Zero-Shot_Grounding_RSNA/inference_attention-map_score.py code served (permissive licence) · get_code("e55695b0b11e1e76") |
| tensor_to_serializable | Not yet run | aiming-lab/mmedpo/inference/utils.py code served (permissive licence) · get_code("af42eb0093076ad3") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The advancement of Large Vision-Language Models (LVLMs) has propelled their application in medicine. However, Medical LVLMs (Med-LVLMs) encounter factuality issues due to modality misalignment, where the models prioritize textual knowledge over visual input, causing hallucinations that conflict with medical images. Previous attempts on preference optimization have inadequately mitigated clinical relevance in preference data, making these samples easily distinguishable and reducing alignment effectiveness. To address this challenge, we propose MMedPO, a novel multimodal medical preference optimization approach that considers the clinical relevance of preference samples to enhance Med-LVLM alignment. MMedPO curates multimodal preference data by introducing two types of dispreference: (1) plausible hallucinations injected through target Med-LVLMs or GPT-4o to produce medically inaccurate responses, and (2) lesion region neglect achieved through local lesion-noising, disrupting visual understanding of critical areas. We then calculate clinical relevance for each sample based on scores from Med-LLMs and visual tools, and integrate these scores into the preference optimization process as weights, enabling effective alignment. Our experiments demonstrate that MMedPO significantly enhances factual accuracy, achieving improvements over existing baseline methods by averaging 14.2% and 51.7% across the Med-VQA and report generation tasks. Our code are available in https://github.com/aiminglab/MMedPO.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2412.06141")
get_code_for_paper("2412.06141")
have("2412.06141")
Connect an agent — have() is free.