SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2103.14517 · ICCV · 2021

On the hidden treasure of dialog in video question answering

Yannis Avrithis, Ngoc Duong, Deniz Engin, Franc ¸ois Schnitzler

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

High-level understanding of stories in video such as movies and TV shows from raw data is extremely challenging. Modern video question answering (VideoQA) systems often use additional human-made sources like plot synopses, scripts, video descriptions or knowledge bases. In this work, we present a new approach to understand the whole story without such external sources. The secret lies in the dialog: unlike any prior work, we treat dialog as a noisy source to be converted into text description via dialog summarization, much like recent methods treat video. The input of each modality is encoded by transformers independently, and a simple fusion method combines all modalities, using soft temporal attention for localization over long inputs. Our model outperforms the state of the art on the KnowIT VQA dataset by a large margin, without using question-specific human annotation or humanmade plot summaries. It even outperforms human evaluators who have never watched any whole episode before. Code is available at https://engindeniz.github. io/dialogsummary-videoqa

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2103.14517")
get_code_for_paper("2103.14517")
have("2103.14517")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2103.14517.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2103.14517.svg)](https://syntology.ai/paper/2103.14517)

Connect an agent — have() is free.