Aren Jansen, Cordelia Schmid, Anurag Arnab, Arsha Nagrani, Shan Yang, Chen Sun
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Humans perceive the world by concurrently processing and fusing highdimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality ('late-fusion') is still a dominant paradigm for multimodal video classification. Instead, we introduce a novel transformer based architecture that uses 'fusion bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense relevant information in each modality and share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released. 35th Conference on Neural Information Processing Systems (NeurIPS 2021).
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2107.00135")
get_code_for_paper("2107.00135")
have("2107.00135")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2107.00135.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2107.00135)
Connect an agent — have() is free.