We lifted 13 functions out of this paper's own repositories and ran 12 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| allenzren/open-pi-zero | pwc_unofficial | 12 of 13 |
| Function | Status | Where it lives |
|---|---|---|
| conditional_decorator | Ran | allenzren/open-pi-zero/src/utils/decorator.py code served (permissive licence) · get_code("d949d0d0bc5e63bc") |
| get_action_accuracy | Ran | allenzren/open-pi-zero/src/utils/metric.py code served (permissive licence) · get_code("7f39687304604d5d") |
| get_num_params_in_billions | Ran | allenzren/open-pi-zero/src/utils/optim.py code served (permissive licence) · get_code("c309b4c8e3254bb9") |
| isrotation | Ran | allenzren/open-pi-zero/src/utils/geometry.py code served (permissive licence) · get_code("ee388693e9bdd485") |
| log_execution_time | Ran | allenzren/open-pi-zero/src/utils/monitor.py code served (permissive licence) · get_code("1c57d6dfeacfc394") |
| lora_state_dict | Ran | allenzren/open-pi-zero/src/model/lora.py code served (permissive licence) · get_code("52614cdfc99cf150") |
| main_rank_only | Ran | allenzren/open-pi-zero/src/utils/decorator.py code served (permissive licence) · get_code("8d01c3eb9b8242f3") |
| mat2euler | Ran | allenzren/open-pi-zero/src/utils/geometry.py code served (permissive licence) · get_code("eed6f0c4289630b1") |
| quat2mat | Ran | allenzren/open-pi-zero/src/utils/geometry.py code served (permissive licence) · get_code("901875b18f961eef") |
| repeat_kv | Ran | allenzren/open-pi-zero/src/model/utils.py code served (permissive licence) · get_code("4e762613ee907cdb") |
| rotate_half | Ran | allenzren/open-pi-zero/src/model/utils.py code served (permissive licence) · get_code("cec833a99feb528c") |
| transpose | Ran | allenzren/open-pi-zero/src/model/lora.py code served (permissive licence) · get_code("67a5b7072d402f6f") |
| apply_rotary_pos_emb | Not yet run | allenzren/open-pi-zero/src/model/utils.py code served (permissive licence) · get_code("354c67f169da36b9") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The development of large language models (LLMs) has expanded to multi-modal systems capable of processing text, images, and speech within a unified framework. Training these models demands significantly larger datasets and computational resources compared to text-only LLMs. To address the scaling challenges, we introduce Mixture-of-Transformers (MoT), a sparse multi-modal transformer architecture that significantly reduces pretraining computational costs. MoT decouples non-embedding parameters of the model by modality -- including feed-forward networks, attention matrices, and layer normalization -- enabling modality-specific processing with global self-attention over the full input sequence. We evaluate MoT across multiple settings and model scales. In the Chameleon 7B setting (autoregressive text-and-image generation), MoT matches the dense baseline's performance using only 55.8\% of the FLOPs. When extended to include speech, MoT reaches speech performance comparable to the dense baseline with only 37.2\% of the FLOPs. In the Transfusion setting, where text and image are trained with different objectives, a 7B MoT model matches the image modality performance of the dense baseline with one third of the FLOPs, and a 760M MoT model outperforms a 1.4B dense baseline across key image generation metrics. System profiling further highlights MoT's practical benefits, achieving dense baseline image quality in 47.2\% of the wall-clock time and text quality in 75.6\% of the wall-clock time (measured on AWS p4de.24xlarge instances with NVIDIA A100 GPUs).
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2411.04996")
get_code_for_paper("2411.04996")
have("2411.04996")
Connect an agent — have() is free.