We lifted 10 functions out of this paper's own repositories and ran 8 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| yifanxu74/libra | canonical | 8 of 10 |
| Function | Status | Where it lives |
|---|---|---|
| apply_rotary_pos_emb | Ran | yifanxu74/libra/libra/models/libra/modeling_libra.py code served (permissive licence) · get_code("45b8a2c070efc3ed") |
| contrastive_loss | Ran | yifanxu74/libra/libra/models/clip/modeling_clip.py code served (permissive licence) · get_code("4675779aec92d6c0") |
| disabled_train | Ran | yifanxu74/libra/libra/models/libra/image_tokenizer.py code served (permissive licence) · get_code("4cb732f513d69dfd") |
| easy_gather | Ran | yifanxu74/libra/libra/models/libra/modeling_libra.py code served (permissive licence) · get_code("7284ea3e763f4e0d") |
| get_encoded_image_size | Ran | yifanxu74/libra/libra/models/libra/tokenization_libra.py code served (permissive licence) · get_code("17279a5f97b07e0d") |
| main_process | Ran | yifanxu74/libra/libra/common/dist_utils.py code served (permissive licence) · get_code("98589643273920ba") |
| node_to_dict | Ran | yifanxu74/libra/libra/common/config.py code served (permissive licence) · get_code("791c72070c0b1cfb") |
| rotate_half | Ran | yifanxu74/libra/libra/models/libra/modeling_libra.py code served (permissive licence) · get_code("b99eea6376d1e212") |
| clip_loss | Not yet run | yifanxu74/libra/libra/models/clip/modeling_clip.py code served (permissive licence) · get_code("f2ca34eca05d40cd") |
| download_cached_file | Not yet run | yifanxu74/libra/libra/common/dist_utils.py code served (permissive licence) · get_code("643539116e049080") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In this work, we introduce Libra, a prototype model with a decoupled vision system on a large language model (LLM). The decoupled vision system decouples inner-modal modeling and cross-modal interaction, yielding unique visual information modeling and effective cross-modal comprehension. Libra is trained through discrete auto-regressive modeling on both vision and language inputs. Specifically, we incorporate a routed visual expert with a cross-modal bridge module into a pretrained LLM to route the vision and language flows during attention computing to enable different attention patterns in inner-modal modeling and cross-modal interaction scenarios. Experimental results demonstrate that the dedicated design of Libra achieves a strong MLLM baseline that rivals existing works in the image-to-text scenario with merely 50 million training data, providing a new perspective for future multimodal foundation models. Code is available at https://github.com/YifanXu74/Libra.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2405.10140")
get_code_for_paper("2405.10140")
have("2405.10140")
Connect an agent — have() is free.