Chang Yoo, Hee Yoon, Eunseop Yoon, Mark Hasegawa-Johnson, Soohwan Eom, Jaehyun Jang
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
While knowledge distillation (KD) is widely adopted for training lightweight models by leveraging supervision from larger teacher models, relying solely on output token distributions has proven insufficient for compressing Multimodal Large Language Models (MLLMs). Since output tokens are a byproduct of the model attending to visual inputs, prior works have explored explicitly distilling attention to provide a direct supervisory signal. While promising, the precise utility of which attention signals to distill remains under-explored. In this work, we challenge the conventional reliance on prompt-to-vision attention by revealing that downstream performance correlates strongly with responseto-vision attention similarity to the teacher, but negligibly with that of prompt-conditioned attention. Furthermore, we observe that attention distributions exhibit significant variance across individual tokens, indicating that a uniform distillation objective is suboptimal. To this end, we introduce Token-level Response-visual Attention Guidance (TRAG), a distillation objective that 1) shifts the focus to responseto-vision signals and 2) employs token-specific objectives by adaptively weighting the Kullback-Leibler divergence based on attention entropy, effectively guiding the student to mirror the teacher's precise visual focus. Extensive experimental results on multiple benchmarks demonstrate that TRAG significantly outperforms prior distillation baselines. Our code is available at https://github.com/jhjangjh/TRAG.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2607.02593")
get_code_for_paper("2607.02593")
have("2607.02593")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2607.02593.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2607.02593)
Connect an agent — have() is free.