SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2404.01476 · EMNLP · 2024

TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answering

Trevor Darrell, Roei Herzig, Amos You, Sanjay Subramanian, Chuyi Shang

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Recently, image-based Large Multimodal Models (LMMs) have made significant progress in video question-answering (VideoQA) using a frame-wise approach by leveraging large-scale pretraining in a zero-shot manner. Nevertheless, these models need to be capable of finding relevant information, extracting it, and answering the question simultaneously. Currently, existing methods perform all of these steps in a single pass without being able to adapt if insufficient or incorrect information is collected. To overcome this, we introduce a modular multi-LMM agent framework based on several agents with different roles, instructed by a Planner agent that updates its instructions using shared feedback from the other agents. Specifically, we propose TraveLER, a method that can create a plan to "Traverse" through the video, ask questions about individual frames to "Locate" and store key information, and then "Evaluate" if there is enough information to answer the question. Finally, if there is not enough information, our method is able to "Replan" based on its collected knowledge. Through extensive experiments, we find that the proposed TraveLER approach improves performance on several VideoQA benchmarks without the need to fine-tune on specific datasets. Our code is available at https://github.com/travelerframework/TraveLER. Choices: (A) train is coming (B) to sit down (C) look at something (D) resting on the yellow object (E) to get down from slide Plan: Go to the middle of the video to find where the boy turns over Q: Is the boy turned on his stomach? Plan: View the frames immediately after to identify any actions Q: Is the boy looking at anything specific? Go Here Q: Do I have enough information to answer the question? A: No Go Here Q: What is the boy doing? Q: Do I have enough information to answer the question? A: No Go Here Q: Is there a train in the background? Q: Is the boy resting? Q: Do I have enough information to answer the question? A: Yes, the answer is (E)

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2404.01476")
get_code_for_paper("2404.01476")
have("2404.01476")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2404.01476.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2404.01476.svg)](https://syntology.ai/paper/2404.01476)

Connect an agent — have() is free.