We lifted 9 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| mc-lan/proxyclip | canonical | 3 of 9 |
| Function | Status | Where it lives |
|---|---|---|
| convert_to_custom_text_state_dict | Ran | mc-lan/proxyclip/open_clip/model.py pointer only (licence: NONE) · get_code("f19962ebb134b3d7") |
| get_cast_dtype | Ran | mc-lan/proxyclip/open_clip/model.py pointer only (licence: NONE) · get_code("dcd422d66b0581d8") |
| get_input_dtype | Ran | mc-lan/proxyclip/open_clip/model.py pointer only (licence: NONE) · get_code("b476c8cfbf0f1f47") |
| gather_features | Not yet run | mc-lan/proxyclip/open_clip/loss.py pointer only (licence: NONE) · get_code("ddcbd45e940484ee") |
| get_cls_idx | Not yet run | mc-lan/proxyclip/proxyclip_segmentor.py pointer only (licence: NONE) · get_code("54731d24b3557840") |
| neighbour_exchange | Not yet run | mc-lan/proxyclip/open_clip/loss.py pointer only (licence: NONE) · get_code("e332856e3c2fc814") |
| neighbour_exchange_bidir | Not yet run | mc-lan/proxyclip/open_clip/loss.py pointer only (licence: NONE) · get_code("5b1fd364afcf3c05") |
| prepare_inputs_for_generation | Not yet run | mc-lan/proxyclip/open_clip/coca_model.py pointer only (licence: NONE) · get_code("fb651d0a97fd4d3f") |
| register_pooler | Not yet run | mc-lan/proxyclip/open_clip/hf_model.py pointer only (licence: NONE) · get_code("2a377da4a76a2d44") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Open-vocabulary semantic segmentation requires models to effectively integrate visual representations with open-vocabulary semantic labels. While Contrastive Language-Image Pre-training (CLIP) models shine in recognizing visual concepts from text, they often struggle with segment coherence due to their limited localization ability. In contrast, Vision Foundation Models (VFMs) excel at acquiring spatially consistent local visual representations, yet they fall short in semantic understanding. This paper introduces ProxyCLIP, an innovative framework designed to harmonize the strengths of both CLIP and VFMs, facilitating enhanced open-vocabulary semantic segmentation. ProxyCLIP leverages the spatial feature correspondence from VFMs as a form of proxy attention to augment CLIP, thereby inheriting the VFMs' robust local consistency and maintaining CLIP's exceptional zero-shot transfer capacity. We propose an adaptive normalization and masking strategy to get the proxy attention from VFMs, allowing for adaptation across different VFMs. Remarkably, as a training-free approach, ProxyCLIP significantly improves the average mean Intersection over Union (mIoU) across eight benchmarks from 40.3 to 44.4, showcasing its exceptional efficacy in bridging the gap between spatial precision and semantic richness for the open-vocabulary segmentation task.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2408.04883")
get_code_for_paper("2408.04883")
have("2408.04883")
Connect an agent — have() is free.