Andrew Bagdanov, Marco Bertini, Marco Mistretta, Alberto Baldrati, Lorenzo Agnolucci
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Pre-trained multi-modal Vision-Language Models like CLIP are widely used offthe-shelf for a variety of applications. In this paper, we show that the common practice of individually exploiting the text or image encoders of these powerful multi-modal models is highly suboptimal for intra-modal tasks like imageto-image retrieval. We argue that this is inherently due to the CLIP-style intermodal contrastive loss that does not enforce any intra-modal constraints, leading to what we call intra-modal misalignment. To demonstrate this, we leverage two optimization-based modality inversion techniques that map representations from their input modality to the complementary one without any need for auxiliary data or additional trained adapters. We empirically show that, in the intra-modal tasks of image-to-image and text-to-text retrieval, approaching these tasks inter-modally significantly improves performance with respect to intramodal baselines on more than fifteen datasets. Additionally, we demonstrate that approaching a native inter-modal task (e.g. zero-shot image classification) intra-modally decreases performance, further validating our findings. Finally, we show that incorporating an intra-modal term in the pre-training objective or narrowing the modality gap between the text and image feature embedding spaces helps reduce the intra-modal misalignment. The code is publicly available at: https://github.com/miccunifi/Cross-the-Gap.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2502.04263")
get_code_for_paper("2502.04263")
have("2502.04263")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2502.04263.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2502.04263)
Connect an agent — have() is free.