Fuzhao Xue, Xinchao Wang, Kai Wang, Yang You, Xiangxiang Chu, Wangbo Zhao
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Text-based video segmentation aims to segment the target object in a video based on a describing sentence. Incorporating motion information from optical flow maps with appearance and linguistic modalities is crucial yet has been largely ignored by previous work. In this paper, we design a method to fuse and align appearance, motion, and linguistic features to achieve accurate segmentation. Specifically, we propose a multi-modal video transformer, which can fuse and aggregate multi-modal and temporal features between frames. Furthermore, we design a language-guided feature fusion module to progressively fuse appearance and motion features in each feature level with guidance from linguistic features. Finally, a multi-modal alignment loss is proposed to alleviate the semantic gap between features from different modalities. Extensive experiments on A2D Sentences and J-HMDB Sentences verify the performance and the generalization ability of our method compared to the state-of-theart methods.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2204.02547")
get_code_for_paper("2204.02547")
have("2204.02547")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2204.02547.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2204.02547)
Connect an agent — have() is free.