We lifted 12 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| jiangsongtao/tinymed | canonical | 5 of 9 |
| jiangsongtao/med-moe | canonical | 1 of 1 |
| copy not recorded | — | 0 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| MoEStablelmDecoderLayer_forward | Ran | jiangsongtao/med-moe/moellava/model/language_model/llava_stablelm_moe.py code served (permissive licence) · get_code("0d5ea0366375f84c") |
| einsum | Ran | jiangsongtao/tinymed/moe/sharded_moe.py code served (permissive licence) · get_code("97286a837fefce75") |
| get_chunk | Ran | jiangsongtao/tinymed/model_vqa_med.py code served (permissive licence) · get_code("42a46570620cd9fa") |
| is_moe_param | Ran | jiangsongtao/tinymed/moe/utils.py code served (permissive licence) · get_code("dee86097e6631190") |
| split_list | Ran | jiangsongtao/tinymed/model_vqa_med.py code served (permissive licence) · get_code("076c252c52cbb161") |
| split_params_into_shared_and_expert_params | Ran | jiangsongtao/tinymed/moe/utils.py code served (permissive licence) · get_code("593d61d6861a857f") |
| gumbel_rsample | Not yet run | jiangsongtao/tinymed/moe/sharded_moe.py code served (permissive licence) · get_code("3b6fa807ed00b1da") |
| has_moe_layers | Not yet run | jiangsongtao/tinymed/moe/utils.py code served (permissive licence) · get_code("d327fe6cbd808182") |
| load_jsonl | Not yet run | jiangsongtao/tinymed/run_eval.py code served (permissive licence) · get_code("77678f2758141df9") |
| multiplicative_jitter | Not yet run | jiangsongtao/tinymed/moe/sharded_moe.py code served (permissive licence) · get_code("765e66c4d56297b6") |
| save_image_to_local | Not yet run | this paper's copy was not recorded; identical code first harvested from PKU-YuanGroup/MoE-LLaVA pointer only · get_code("e2e5c3a95a6aebb4") |
| save_video_to_local | Not yet run | this paper's copy was not recorded; identical code first harvested from PKU-YuanGroup/MoE-LLaVA pointer only · get_code("22930d58d32e2d08") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent advancements in general-purpose or domain-specific multimodal large language models (LLMs) have witnessed remarkable progress for medical decision-making. However, they are designated for specific classification or generative tasks, and require model training or finetuning on large-scale datasets with sizeable parameters and tremendous computing, hindering their clinical utility across diverse resource-constrained scenarios in practice. In this paper, we propose a novel and lightweight framework Med-MoE (Mixture-of-Experts) that tackles both discriminative and generative multimodal medical tasks. The learning of Med-MoE consists of three steps: multimodal medical alignment, instruction tuning and routing, and domain-specific MoE tuning. After aligning multimodal medical images with LLM tokens, we then enable the model for different multimodal medical tasks with instruction tuning, together with a trainable router tailored for expert selection across input modalities. Finally, the model is tuned by integrating the router with multiple domain-specific experts, which are selectively activated and further empowered by meta expert. Comprehensive experiments on both open- and close-end medical question answering (Med-VQA) and image classification tasks across datasets such as VQA-RAD, SLAKE and Path-VQA demonstrate that our model can achieve performance superior to or on par with state-of-the-art baselines, while only requiring approximately 30\%-50\% of activated model parameters. Extensive analysis and ablations corroborate the effectiveness and practical utility of our method.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2404.10237")
get_code_for_paper("2404.10237")
have("2404.10237")
Connect an agent — have() is free.