We lifted 7 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| ola-omni/ola | canonical | 3 of 7 |
| Function | Status | Where it lives |
|---|---|---|
| pad_image | Ran | ola-omni/ola/ola/mm_utils.py code served (permissive licence) · get_code("1aa848dda4bdce79") |
| resize_images | Ran | ola-omni/ola/ola/mm_utils.py code served (permissive licence) · get_code("6b0f79b82970ce8b") |
| resize_video | Ran | ola-omni/ola/ola/mm_utils.py code served (permissive licence) · get_code("5ba000d6ad89a790") |
| build_unit_tokenizer | Not yet run | ola-omni/ola/ola/serve/model_worker.py code served (permissive licence) · get_code("0746bf62da5b6d97") |
| build_vision_tower | Not yet run | ola-omni/ola/ola/model/multimodal_encoder/builder.py code served (permissive licence) · get_code("3b3e8a53bce0fb0f") |
| get_peft_state_maybe_zero_3 | Not yet run | ola-omni/ola/ola/utils.py code served (permissive licence) · get_code("c5550dd05cf0b148") |
| maybe_zero_3 | Not yet run | ola-omni/ola/ola/utils.py code served (permissive licence) · get_code("720e6dbf9d7eb553") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent advances in large language models, particularly following GPT-4o, have sparked increasing interest in developing omni-modal models capable of understanding more modalities. While some open-source alternatives have emerged, there is still a notable lag behind specialized single-modality models in performance. In this paper, we present Ola, an Omni-modal Language model that achieves competitive performance across image, video, and audio understanding compared to specialized counterparts, pushing the frontiers of the omni-modal language model to a large extent. We conduct a comprehensive exploration of architectural design, data curation, and training strategies essential for building a robust omni-modal model. Ola incorporates advanced visual understanding and audio recognition capabilities through several critical and effective improvements over mainstream baselines. Moreover, we rethink inter-modal relationships during omni-modal training, emphasizing cross-modal alignment with video as a central bridge, and propose a progressive training pipeline that begins with the most distinct modalities and gradually moves towards closer modality alignment. Extensive experiments demonstrate that Ola surpasses existing open omni-modal LLMs across all modalities while achieving highly competitive performance compared to state-of-the-art specialized models of similar sizes. We aim to make Ola a fully open omni-modal understanding solution to advance future research in this emerging field. Model weights, code, and data are open-sourced at https://github.com/Ola-Omni/Ola.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2502.04328")
get_code_for_paper("2502.04328")
have("2502.04328")
Connect an agent — have() is free.