Ziwei Liu, Lewei Lu, Penghao Wu
We lifted 9 functions out of this paper's own repositories and ran 5 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| penghao-wu/proxyv | canonical | 1 of 5 |
| copy not recorded | — | 4 of 4 |
| Function | Status | Where it lives |
|---|---|---|
| FeedForward | Ran | penghao-wu/proxyv/llava/model/multimodal_resampler/perceiver.py code served (permissive licence) · get_code("4d75570b4e91d0b4") |
| convert_to_custom_text_state_dict | Ran | this paper's copy was not recorded; identical code first harvested from ToTheBeginning/PuLID pointer only · get_code("0a65604f002e6ce4") |
| convert_to_custom_text_state_dict | Ran | this paper's copy was not recorded; identical code first harvested from zsxkib/PuLID pointer only · get_code("7bff6f3b3cb560e7") |
| get_cast_dtype | Ran | this paper's copy was not recorded; identical code first harvested from ToTheBeginning/PuLID pointer only · get_code("61d9ad9efd32efee") |
| get_cast_dtype | Ran | this paper's copy was not recorded; identical code first harvested from zsxkib/PuLID pointer only · get_code("dcd422d66b0581d8") |
| PerceiverAttention | Not yet run | penghao-wu/proxyv/llava/model/multimodal_resampler/perceiver.py code served (permissive licence) · get_code("6a96fbd946c4017a") |
| PerceiverResampler | Not yet run | penghao-wu/proxyv/llava/model/multimodal_resampler/perceiver.py code served (permissive licence) · get_code("98b4bae0261b6b65") |
| PerceiverResamplerModule | Not yet run | penghao-wu/proxyv/llava/model/multimodal_resampler/perceiver.py code served (permissive licence) · get_code("930ebfa80c5b9f00") |
| build_model_from_openai_state_dict | Not yet run | penghao-wu/proxyv/llava/model/multimodal_encoder/dev_eva_clip/eva_clip/model.py code served (permissive licence) · get_code("a8e9981d9caacade") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large multimodal models excel in multimodal tasks but face significant computational challenges due to excessive computation on visual tokens. Unlike token reduction methods that focus on token-level redundancy, we identify and study the computation-level redundancy on vision tokens to ensure no information loss. Our key insight is that vision tokens from the pretrained vision encoder do not necessarily require all the heavy operations (e.g., self-attention, FFNs) in decoder-only LMMs and could be processed more lightly with proper designs. We designed a series of experiments to discover and progressively squeeze out the vision-related computation redundancy. Based on our findings, we propose ProxyV, a novel approach that utilizes proxy vision tokens to alleviate the computational burden on original vision tokens. ProxyV enhances efficiency without compromising performance and can even yield notable performance gains in scenarios with more moderate efficiency improvements. Furthermore, the flexibility of ProxyV is demonstrated through its combination with token reduction methods to boost efficiency further. The code will be made public here.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2505.15816")
get_code_for_paper("2505.15816")
have("2505.15816")
Connect an agent — have() is free.