SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models

Hongsheng Li, Wenqi Shao, Siyuan Huang, Chao Xu, Yu Qiao, Renrui Zhang, Peng Gao, Pan Lu, Hao Shao, Junjun He, Ziyi Lin, Shijie Geng, and 7 more

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repository	Role	Ran
alpha-vllm/llama2-accessory	canonical	3 of 3

Function	Status	Where it lives
apply_rotary_emb	Ran	alpha-vllm/llama2-accessory/accessory/model/LLM/llama.py pointer only (licence: NOASSERTION) · get_code("dcae39c8928b5fd8")
precompute_freqs_cis	Ran	alpha-vllm/llama2-accessory/accessory/model/LLM/llama.py pointer only (licence: NOASSERTION) · get_code("2c4423db8989ee05")
reshape_for_broadcast	Ran	alpha-vllm/llama2-accessory/accessory/model/LLM/llama.py pointer only (licence: NOASSERTION) · get_code("70bf6ebaafd266c4")

Repositories linked to this paper

alpha-vllm/llama2-accessory — marked official in Papers with Code; named in the paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM) series developed upon SPHINX. To improve the architecture and training efficiency, we modify the SPHINX framework by removing redundant visual encoders, bypassing fully-padded sub-images with skip tokens, and simplifying multi-stage training into a one-stage all-in-one paradigm. To fully unleash the potential of MLLMs, we assemble a comprehensive multi-domain and multimodal dataset covering publicly available resources in language, vision, and vision-language tasks. We further enrich this collection with our curated OCR intensive and Set-of-Mark datasets, extending the diversity and generality. By training over different base LLMs including TinyLlama-1.1B, InternLM2-7B, LLaMA2-13B, and Mixtral-8×7B, we obtain a spectrum of MLLMs that vary in parameter size and multilingual capabilities. Comprehensive benchmarking reveals a strong correlation between the multi-modal performance with the data and parameter scales. Code and models are released at https://github.com/ Alpha-VLLM/LLaMA2-Accessory.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2402.05935")
get_code_for_paper("2402.05935")
have("2402.05935")

Connect an agent — have() is free.