BAT: Learning to Reason about Spatial Sounds with Large Language Models

1The University of Texas at Austin, 2Shanghai Jiao Tong University
ICML 2024

Abstract

Spatial sound reasoning is a fundamental human skill, enabling us to navigate and interpret our surroundings based on sound. In this paper we present BAT, which combines the spatial sound perception ability of a binaural acoustic scene analysis model with the natural language reasoning capabilities of a large language model (LLM) to replicate this innate ability. To address the lack of existing datasets of in-the-wild spatial sounds, we synthesized a binaural audio dataset using AudioSet and SoundSpaces 2.0. Next, we developed SpatialSoundQA, a spatial sound-based question-answering dataset, offering a range of QA tasks that train BAT in various aspects of spatial sound perception and reasoning. The acoustic front end encoder of BAT is a novel spatial audio encoder named Spatial Audio Spectrogram Transformer, or Spatial-AST, which by itself achieves strong performance across sound event detection, spatial localization, and distance estimation. By integrating Spatial-AST with LLaMA-2 7B model, BAT transcends standard Sound Event Localization and Detection (SELD) tasks, enabling the model to reason about the relationships between the sounds in its environment. Our experiments demonstrate BAT’s superior performance on both spatial sound perception and reasoning, showcasing the immense potential of LLMs in navigating and interpreting complex spatial audio environments.

BAT architecture

Interpolate start reference image.

BAT model structure. The rightmost part shows the binaural audio generation process.

<

SpatialSoundQA

Type
#Sources
Example
A: Detection
(139K, 15.9%)
1
Q: Identify the sound events in the audio clip. / A: baby laughter; laughter; speech
Q: What are the distinct sounds present in this audio clip? / A: heart sounds, heartbeat
B: DoA & DP
(139K, 15.9%)
1
Q: How would you describe the location of this audio clip? / A: right, front, below; 2.5m
Q: At what distance and in which direction, is the music’s sound originating? / A: left, behind, below; 5m
C: Detection
(118K, 13.5%)
2
Q: Identify the sound events in the audio clip coming from the right, front, below, approximately 3 meters away. / A: slosh; speech
Q: What sound events can you detect in the audio recording emanating from the left, behind, above, roughly 0.5 meters away?
A: music; musical instrument; steelpan
D: DoA & DP
(118K, 13.5%)
2
Q: In which direction and how far away is the source of the heart sounds, heartbeat’s sound? / A: left, behind, below; 1m
Q: Where is the sound of the music coming from? / A: left, behind, below; 3m
E: Reasoning
(358K, 41.2%)
2
Q: When measuring the direct line distance, is the sound produced by wheeze closer to you than the sound from bird flight, flapping wings? / A: No
Q: Is the source of both explosion and speech’s sounds from your left side? / A: Yes
Q: Is the sound from the music on the front and the sound from the rattle on the behind? / A: No
Q: Does the sound of electric shaver, electric razor appear on the behind of the sound of waterfall? / A: Yes
Q: Can you estimate the distance from the sound of the speech to the sound of the dog? / A: 1.5m
Q: What is the sound on the above side of the sound of the vibration? / A: croak; frog
Q: Could you determine whether the singing’s sound is to the left or right of the steam’s sound? / A: left

Distance Perception

( Please wear your headphones!)

Clean dog barking
-4
-2
0
2
4
Receiver
Distance/m

Question Type A: Detection

Q: Enumerate the sound occurrences in the audio clip.
A: basketball bounce; music
Q: Can you categorize the sounds in the audio?
A: engine; rattle; vehicle

Question Type B: Distance Prediction & DoA

Q: In your opinion, where is this audio clip coming from?
A: right, front, below; 0.5m
Q: Where is the audio clip coming from?
A: left, front, below; 4m

Question Type C: Mix-up Detection

Q: What sound events can you detect in the audio recording emanating from
the right,front, below, roughly 3.5 meters away?
A: ambulance (siren); emergency vehicle; siren
Q: Determine the types of sounds present in the audio clip from directions to
the left, front, above, approximately 2 meters distant.
A: explosion
Hint: Another sound source: rumble

Question Type D: Mix-up Distance Prediction & DoA

Q: How would you describe the location of the crow's sound?
A: left, behind, below; 3m
Q: Where is the sound of the mechanical fan?
A: left, behind, below; 3m

Question Type E: Reasoning

Q: Does the sound of gunshot, gunfire appear on the left of the sound of speech?
A: Yes
Q: Is the sound from cowbell further away from you in Euclidean distance than
the sound from music?
A: No

BibTeX

@article{zheng2024bat,
  author    = {Zheng, Zhisheng and Peng, Puyuan and Ma, Ziyang and Chen, Xie and Choi, Eunsol and Harwath, David},
  title     = {BAT: Learning to Reason about Spatial Sounds with Large Language Models},
  journal   = {arXiv preprint arXiv:2402.01591},
  year      = {2024},
}