Is Space-Time Attention All You Need for Video Understanding?

Gedas Bertasius, Lorenzo Torresani, Heng Wang

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 43 functions out of this paper's own repositories and ran 35 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• facebookresearch/TimeSformer — marked official in Papers with Code; named in the paper
• davide-coccomini/TimeSformer-Video-Classification
• halixness/generative_timesformer_pytorch
• jerrywn121/TianChi_AIEarth
• lucidrains/TimeSformer-pytorch
• m-bain/video-transformers
• md-mohaiminul/objectstatechange
• mx-mark/videotransformer-pytorch
• yiyixuxu/TimeSformer-rolled-attention
• MindCode-4/code-5
• PaddlePaddle/PaddleVideo
• The-AI-Summer/self-attention-cv
• open-mmlab/mmaction2
• pwc-1/Paper-10
• pwc-1/Paper-9
• towhee-io/towhee

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We present a convolution-free approach to video classification built exclusively on self-attention over space and time. Our method, named "TimeSformer," adapts the standard Transformer architecture to video by enabling spatiotemporal feature learning directly from a sequence of framelevel patches. Our experimental study compares different self-attention schemes and suggests that "divided attention," where temporal attention and spatial attention are separately applied within each block, leads to the best video classification accuracy among the design choices considered. Despite the radically new design, TimeSformer achieves state-of-the-art results on several action recognition benchmarks, including the best reported accuracy on Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks, our model is faster to train, it can achieve dramatically higher test efficiency (at a small drop in accuracy), and it can also be applied to much longer video clips (over one minute long). Code and models are available at: https://github.com/ facebookresearch/TimeSformer.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2102.05095")
get_code_for_paper("2102.05095")
have("2102.05095")

Connect an agent — have() is free.