Yunchao Wei, Yao Zhao, Siyu Jiao, Humphrey Shi, Hongguang Zhu, Jiannan Huang
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Pre-trained vision-language models, e.g. CLIP, have been increasingly used to address the challenging Open-Vocabulary Segmentation (OVS) task, benefiting from their well-aligned vision-text embedding space. Typical solutions involve either freezing CLIP during training to unilaterally maintain its zero-shot capability, or fine-tuning CLIP vision encoder to achieve perceptual sensitivity to local regions. However, few of them incorporate vision-text collaborative optimization. Based on this, we propose the Content-Dependent Transfer to adaptively enhance each text embedding by interacting with the input image, which presents a parameter-efficient way to optimize the text representation. Besides, we additionally introduce a Representation Compensation strategy, reviewing the original CLIP-V representation as compensation to maintain the zero-shot capability of CLIP. In this way, the vision and text representation of CLIP are optimized collaboratively, enhancing the alignment of the vision-text feature space. To the best of our knowledge, we are the first to establish the collaborative vision-text optimizing mechanism within the OVS field. Extensive experiments demonstrate our method achieves superior performance on popular OVS benchmarks. In openvocabulary semantic segmentation, our method outperforms the previous state-of-the-art approaches by +0.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2408.00744")
get_code_for_paper("2408.00744")
have("2408.00744")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2408.00744.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2408.00744)
Connect an agent — have() is free.