SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2006.16228 · NeurIPS · 2020

Self-Supervised MultiModal Versatile Networks

Andrew Zisserman, Rosalia Schneider, Sander Dieleman, Jean-Baptiste Alayrac, Adrià Recasens, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visual, audio and language streams. To this end, we introduce the notion of a multimodal versatile network -a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of the visual and audio modalities can be maintained, whilst also integrating text into a common embedding. Driven by versatility, we also introduce a novel process of deflation, so that the networks can be effortlessly applied to the visual data in the form of video or a static image. We demonstrate how such networks trained on large collections of unlabelled video data can be applied on video, video-text, image and audio tasks. Equipped with these representations, we obtain state-of-the-art performance on multiple challenging benchmarks including UCF101, HMDB51, Kinetics600, Audioset and ESC-50 when compared to previous self-supervised work. Our models are publicly available [1,2,3].

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2006.16228")
get_code_for_paper("2006.16228")
have("2006.16228")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2006.16228.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2006.16228.svg)](https://syntology.ai/paper/2006.16228)

Connect an agent — have() is free.