Yi Li, Nuno Vasconcelos, Subarna Tripathi, Kyle Min
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Do video-text transformers learn to model temporal relationships across frames? Despite their immense capacity and the abundance of multimodal training data, recent work has revealed the strong tendency of video-text models towards frame-based spatial representations, while temporal reasoning remains largely unsolved. In this work, we identify several key challenges in temporal learning of videotext transformers: the spatiotemporal trade-off from limited network size; the curse of dimensionality for multi-frame modeling; and the diminishing returns of semantic information by extending clip length. Guided by these findings, we propose SViTT, a sparse video-text architecture that performs multi-frame reasoning with significantly lower cost than naïve transformers with dense attention. Analogous to graph-based networks, SViTT employs two forms of sparsity: edge sparsity that limits the query-key communications between tokens in self-attention, and node sparsity that discards uninformative visual tokens. Trained with a curriculum which increases model sparsity with the clip length, SViTT outperforms dense transformer baselines on multiple video-text retrieval and question answering benchmarks, with a fraction of computational cost. Project
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2304.08809")
get_code_for_paper("2304.08809")
have("2304.08809")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2304.08809.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2304.08809)
Connect an agent — have() is free.