Liang Zhao, Xiangyu Zhang, Li, Jinrong Yang, Runpei Dong, Kaisheng Ma, Zekun Qi, Jianjian Sun, Zheng Ge, Xiangwen Kong, Chunrui Han, Yuang Peng, and 2 more
We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| RunpeiDong/DreamLLM | canonical | 1 of 1 |
| copy not recorded | — | 2 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| repeat_kv | Ran | this paper's copy was not recorded; identical code first harvested from fe1ixxu/ALMA pointer only · get_code("30d7eec482ebf6b1") |
| apply_rotary_pos_emb | Ran | RunpeiDong/DreamLLM/omni/models/dreamllm/modeling_dreamllm.py code served (permissive licence) · get_code("d61c483a3c2b3156") |
| rotate_half | Ran | this paper's copy was not recorded; identical code first harvested from fe1ixxu/ALMA pointer only · get_code("b99eea6376d1e212") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
This paper presents DREAMLLM, a learning framework that first achieves versatile Multimodal Large Language Models (MLLMs) empowered with frequently overlooked synergy between multimodal comprehension and creation. DREAMLLM operates on two fundamental principles. The first focuses on the generative modeling of both language and image posteriors by direct sampling in the raw multimodal space. This approach circumvents the limitations and information loss inherent to external feature extractors like CLIP, and a more thorough multimodal understanding is obtained. Second, DREAMLLM fosters the generation of raw, interleaved documents, modeling both text and image contents, along with unstructured layouts. This allows DREAMLLM to learn all conditional, marginal, and joint multimodal distributions effectively. As a result, DREAMLLM is the first MLLM capable of generating free-form interleaved content. Comprehensive experiments highlight DREAMLLM's superior performance as a zero-shot multimodal generalist, reaping from the enhanced learning synergy. Project page: dreamllm.github.io. "What I cannot create, I do not understand." Richard P. Feynman, on his blackboard at the time of his death, 1988
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2309.11499")
get_code_for_paper("2309.11499")
have("2309.11499")
Connect an agent — have() is free.