SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2402.01591 · ICML · 2024

BAT: Learning to Reason about Spatial Sounds with Large Language Models

Xie Chen, Puyuan Peng, Eunsol Choi, David Harwath, Zhisheng Zheng, Ziyang Ma

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.

Abstract

Spatial sound reasoning is a fundamental human skill, enabling us to navigate and interpret our surroundings based on sound. In this paper we present BAT, which combines the spatial sound perception ability of a binaural acoustic scene analysis model with the natural language reasoning capabilities of a large language model (LLM) to replicate this innate ability. To address the lack of existing datasets of in-the-wild spatial sounds, we synthesized a binaural audio dataset using Au-dioSet and SoundSpaces 2.0. Next, we developed SPATIALSOUNDQA, a spatial sound-based question-answering dataset, offering a range of QA tasks that train BAT in various aspects of spatial sound perception and reasoning. The acoustic front end encoder of BAT is a novel spatial audio encoder named Spatial Audio Spectrogram Transformer, or SPATIAL-AST, which by itself achieves strong performance across sound event detection, spatial localization, and distance estimation. By integrating SPATIAL-AST with LLaMA-2 7B model, BAT transcends standard Sound Event Localization and Detection (SELD) tasks, enabling the model to reason about the relationships between the sounds in its environment. Our experiments demonstrate BAT's superior performance on both spatial sound perception and reasoning, showcasing the immense potential of LLMs in navigating and interpreting complex spatial audio environments. Our demo, dataset, code and model weights are available at: https://zhishengzheng.com/bat.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2402.01591")
get_code_for_paper("2402.01591")
have("2402.01591")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2402.01591.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2402.01591.svg)](https://syntology.ai/paper/2402.01591)

Connect an agent — have() is free.