SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2108.05565 · ICCV · 2021

Vision-Language Transformer and Query Generation for Referring Segmentation

Chang Liu, Henghui Ding, Xudong Jiang, Suchen Wang

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

In this work, we address the challenging task of referring segmentation. The query expression in referring segmentation typically indicates the target object by describing its relationship with others. Therefore, to find the target one among all instances in the image, the model must have a holistic understanding of the whole image. To achieve this, we reformulate referring segmentation as a direct attention problem: finding the region in the image where the query language expression is most attended to. We introduce transformer and multi-head attention to build a network with an encoder-decoder attention mechanism architecture that "queries" the given image with the language expression. Furthermore, we propose a Query Generation Module, which produces multiple sets of queries with different attention weights that represent the diversified comprehensions of the language expression from different aspects. At the same time, to find the best way from these diversified comprehensions based on visual clues, we further propose a Query Balance Module to adaptively select the output features of these queries for a better mask generation. Without bells and whistles, our approach is light-weight and achieves new state-of-theart performance consistently on three referring segmentation datasets, RefCOCO, RefCOCO+, and G-Ref. Our code is available at https://github.com/henghuiding/Vision-Language-Transformer. Decode "Small elephant on the left" Vision-Guided Attention 0.6 0.7 0.3 Q1 "SMALL elephant on the LEFT" Q2 "SMALL ELEPHANT on the left" Q3 "small ELEPHANT on the LEFT"

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2108.05565")
get_code_for_paper("2108.05565")
have("2108.05565")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2108.05565.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2108.05565.svg)](https://syntology.ai/paper/2108.05565)

Connect an agent — have() is free.