Yu Qiao, Weidi Xie, Rui Feng, Yuejie Zhang, Yi Wang, Jilan Xu, Junlin Hou
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
This paper considers the problem of open-vocabulary semantic segmentation (OVS), that aims to segment objects of arbitrary classes beyond a pre-defined, closed-set categories. The main contributions are as follows: First, we propose a transformer-based model for OVS, termed as OVSegmentor, which only exploits web-crawled imagetext pairs for pre-training without using any mask annotations. OVSegmentor assembles the image pixels into a set of learnable group tokens via a slot-attention based binding module, then aligns the group tokens to corresponding caption embeddings. Second, we propose two proxy tasks for training, namely masked entity completion and cross-image mask consistency. The former aims to infer all masked entities in the caption given group tokens, that enables the model to learn fine-grained alignment between visual groups and text entities. The latter enforces consistent mask predictions between images that contain shared entities, encouraging the model to learn visual invariance. Third, we construct CC4M dataset for pre-training by filtering CC12M with frequently appeared entities, which significantly improves training efficiency. Fourth, we perform zero-shot transfer on four benchmark datasets, PAS-CAL VOC, PASCAL Context, COCO Object, and ADE20K. OVSegmentor achieves superior results over state-of-theart approaches on PASCAL VOC using only 3% data (4M vs 134M) for pre-training.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2301.09121")
get_code_for_paper("2301.09121")
have("2301.09121")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2301.09121.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2301.09121)
Connect an agent — have() is free.