Gaurav Verma, Kartik Sharma, Srijan Kumar, Minje Choi, Jamelle Watson-Daniels, Sejoon Oh
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Multimodal large language models (MLLMs) like LLaVA and GPT-4(V) enable generalpurpose conversations about images with the language modality. As off-the-shelf MLLMs may have limited capabilities on images from domains like dermatology and agriculture, they must be fine-tuned to unlock domain-specific applications. The prevalent architecture of current open-source MLLMs comprises two major modules: an image-language (cross-modal) projection network and a large language model. It is desirable to understand the roles of these two modules in modeling domain-specific visual attributes to inform the design of future models and streamline the interpretability efforts on the current models. To this end, via experiments on 4 datasets and under 2 fine-tuning settings, we find that as the MLLM is finetuned, it indeed gains domain-specific visual capabilities, but the updates do not lead to the projection extracting relevant domain-specific visual attributes. Our results indicate that the domain-specific visual attributes are modeled by the LLM, even when only the projection is fine-tuned. Through this study, we offer a potential reinterpretation of the role of crossmodal projections in MLLM architectures.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2402.16832")
get_code_for_paper("2402.16832")
have("2402.16832")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2402.16832.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2402.16832)
Connect an agent — have() is free.