SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2107.00135 · NeurIPS · 2021

Attention Bottlenecks for Multimodal Fusion

Aren Jansen, Cordelia Schmid, Anurag Arnab, Arsha Nagrani, Shan Yang, Chen Sun

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Humans perceive the world by concurrently processing and fusing highdimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality ('late-fusion') is still a dominant paradigm for multimodal video classification. Instead, we introduce a novel transformer based architecture that uses 'fusion bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense relevant information in each modality and share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released. 35th Conference on Neural Information Processing Systems (NeurIPS 2021).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2107.00135")
get_code_for_paper("2107.00135")
have("2107.00135")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2107.00135.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2107.00135.svg)](https://syntology.ai/paper/2107.00135)

Connect an agent — have() is free.