Chun Yuan, Jiaqi Wang, Dachuan Shi, Chaofan Tao, Zhendong Yang, Anyi Rao, Attn Ffn, Attn
We lifted 4 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| sdc17/CrossGET | canonical | 2 of 2 |
| sdc17/crossget | — | 0 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| evaluation | Ran | sdc17/CrossGET/CLIP/train_retrieval_clip.py code served (permissive licence) · get_code("d1e13682bf9d66ff") |
| itm_eval | Ran | sdc17/CrossGET/CLIP/train_retrieval_clip.py code served (permissive licence) · get_code("c7f85279d772ea19") |
| ResidualAttentionBlock | Not yet run | sdc17/crossget/CLIP/clip/model.py code served (permissive licence) · get_code("5564a2cac36e0f33") |
| TokenReduction | Not yet run | sdc17/crossget/CLIP/clip/model.py code served (permissive licence) · get_code("2af83242ad1e7b11") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent vision-language models have achieved tremendous advances. However, their computational costs are also escalating dramatically, making model acceleration exceedingly critical. To pursue more efficient vision-language Transformers, this paper introduces Cross-Guided Ensemble of Tokens (CrossGET), a general acceleration framework for vision-language Transformers. This framework adaptively combines tokens in real-time during inference, significantly reducing computational costs while maintaining high performance. CrossGET features two primary innovations: 1) Cross-Guided Matching and Ensemble. CrossGET leverages cross-modal guided token matching and ensemble to effectively utilize cross-modal information, achieving wider applicability across both modality-independent models, e.g., CLIP, and modality-dependent ones, e.g., BLIP2. 2) Complete-Graph Soft Matching. CrossGET introduces an algorithm for the token-matching mechanism, ensuring reliable matching results while facilitating parallelizability and high efficiency. Extensive experiments have been conducted on various vision-language tasks, such as image-text retrieval, visual reasoning, image captioning, and visual question answering. The performance on both classic multimodal architectures and emerging multimodal LLMs demonstrates the framework's effectiveness and versatility. The code is available at https://github.com/sdc17/CrossGET.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2305.17455")
get_code_for_paper("2305.17455")
have("2305.17455")
Connect an agent — have() is free.