Bo Zhu, Li Yuan, Jin Peng, Shuicheng Yan
We lifted 18 functions out of this paper's own repositories and ran 14 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| pku-yuangroup/chat-univi | canonical | 3 of 5 |
| SkyworkAI/MoH | canonical | 3 of 3 |
| skyworkai/moe-plus-plus | — | 5 of 6 |
| copy not recorded | — | 3 of 3 |
| skyworkai/moh | — | 0 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| normal_kl | Ran | SkyworkAI/MoH/MoH-DiT/diffusion/diffusion_utils.py code served (permissive licence) · get_code("3f37a54ade181a61") |
| repeat_kv | Ran | this paper's copy was not recorded; identical code first harvested from fe1ixxu/ALMA pointer only · get_code("30d7eec482ebf6b1") |
| DynamicNTKScalingRotaryEmbedding | Ran | skyworkai/moe-plus-plus/MoE++/modeling_moe_plus_plus.py code served (permissive licence) · get_code("c39d2f7f60570686") |
| LinearScalingRotaryEmbedding | Ran | skyworkai/moe-plus-plus/MoE++/modeling_moe_plus_plus.py code served (permissive licence) · get_code("eebf71aed18a0f87") |
| MoeConfig | Ran | skyworkai/moe-plus-plus/MoE++/modeling_moe_plus_plus.py code served (permissive licence) · get_code("420932dc6224ccb9") |
| NTKScalingRotaryEmbedding | Ran | skyworkai/moe-plus-plus/MoE++/modeling_moe_plus_plus.py code served (permissive licence) · get_code("9a1f7fea1bda87f8") |
| RotaryEmbedding | Ran | skyworkai/moe-plus-plus/MoE++/modeling_moe_plus_plus.py code served (permissive licence) · get_code("4390a52a532857aa") |
| apply_rotary_pos_emb | Ran | this paper's copy was not recorded; identical code first harvested from RunpeiDong/DreamLLM pointer only · get_code("d61c483a3c2b3156") |
| approx_standard_normal_cdf | Ran | SkyworkAI/MoH/MoH-DiT/diffusion/diffusion_utils.py code served (permissive licence) · get_code("d6a68e210556f857") |
| continuous_gaussian_log_likelihood | Ran | SkyworkAI/MoH/MoH-DiT/diffusion/diffusion_utils.py code served (permissive licence) · get_code("57848f6ed380e2e5") |
| get_chunk | Ran | pku-yuangroup/chat-univi/ChatUniVi/eval/model_coco_vqa.py code served (permissive licence) · get_code("42a46570620cd9fa") |
| read_json | Ran | pku-yuangroup/chat-univi/ChatUniVi/eval/model_video_qa.py code served (permissive licence) · get_code("c578a6846674ff45") |
| rotate_half | Ran | this paper's copy was not recorded; identical code first harvested from fe1ixxu/ALMA pointer only · get_code("b99eea6376d1e212") |
| split_list | Ran | pku-yuangroup/chat-univi/ChatUniVi/eval/model_coco_vqa.py code served (permissive licence) · get_code("076c252c52cbb161") |
| Attention | Not yet run | skyworkai/moe-plus-plus/MoE++/modeling_moe_plus_plus.py code served (permissive licence) · get_code("513d27b0995b5f3e") |
| Attention | Not yet run | skyworkai/moh/MoH-ViT/attention.py code served (permissive licence) · get_code("7b9f5a8e366cc24e") |
| save_image_to_local | Not yet run | pku-yuangroup/chat-univi/main_demo_13B.py code served (permissive licence) · get_code("e2e5c3a95a6aebb4") |
| save_video_to_local | Not yet run | pku-yuangroup/chat-univi/main_demo_13B.py code served (permissive licence) · get_code("cab3546afd60006d") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In this work, we upgrade the multi-head attention mechanism, the core of the Transformer model, to reduce computational costs while maintaining or surpassing the previous accuracy level. We show that multi-head attention can be expressed in the summation form. Drawing on the insight that not all attention heads hold equal significance, we propose Mixture-of-Head attention (MoH), a new architecture that treats attention heads as experts in the Mixture-of-Experts (MoE) mechanism. MoH has two significant advantages: First, MoH enables each token to select the appropriate attention heads, enhancing inference efficiency without compromising accuracy or increasing the number of parameters. Second, MoH replaces the standard summation in multi-head attention with a weighted summation, introducing flexibility to the attention mechanism and unlocking extra performance potential. Extensive experiments on ViT, DiT, and LLMs demonstrate that MoH outperforms multi-head attention by using only 50%∼90% of the attention heads. Moreover, we demonstrate that pre-trained multi-head attention models, such as LLaMA3-8B, can be further continue-tuned into our MoH models. Notably, MoH-LLaMA3-8B achieves an average accuracy of 64.0% across 14 benchmarks, outperforming LLaMA3-8B by 2.4% by utilizing only 75% of the attention heads. We believe the proposed MoH is a promising alternative to multi-head attention and provides a strong foundation for developing advanced and efficient attention-based models. The code is available at
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2410.11842")
get_code_for_paper("2410.11842")
have("2410.11842")
Connect an agent — have() is free.