We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The thud of a bouncing ball, the onset of speech as lips open -- when visual and audio events occur together, it suggests that there might be a common, underlying event that produced both signals. In this paper, we argue that the visual and audio components of a video signal should be modeled jointly using a fused multisensory representation. We propose to learn such a representation in a self-supervised way, by training a neural network to predict whether video frames and audio are temporally aligned. We use this learned representation for three applications: (a) sound source localization, i.e. visualizing the source of sound in a video; (b) audio-visual action recognition; and (c) on/off-screen audio source separation, e.g. removing the off-screen translator's voice from a foreign official's speech. Code, models, and video results are available on our webpage: http://andrewowens.com/multisensory
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("1804.03641")
get_code_for_paper("1804.03641")
have("1804.03641")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/1804.03641.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/1804.03641)
Connect an agent — have() is free.