Interpretable Perception and Reasoning for Audiovisual Geolocation

• URL: http://arxiv.org/abs/2603.05708v1
• Date: Thu, 05 Mar 2026 22:04:58 GMT
• Title: Interpretable Perception and Reasoning for Audiovisual Geolocation
• Abstract summary: We introduce Audiovisual Geolocation, a framework designed to resolve geographic ambiguity through interpretable perception and reasoning.<n>We present AVG, a high-quality global-scale video benchmark for geolocation, comprising 20,000 curated clips across 1,000 distinct locations.
• Score: 13.223409217215648
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: While recent advances in Multimodal Large Language Models (MLLMs) have improved image-based localization, precise global geolocation remains a formidable challenge due to the inherent ambiguity of visual landscapes and the largely untapped potential of auditory cues. In this paper, we introduce Audiovisual Geolocation, a framework designed to resolve geographic ambiguity through interpretable perception and reasoning. We present AVG, a high-quality global-scale video benchmark for geolocation, comprising 20,000 curated clips across 1,000 distinct locations. To address the complexity of audiovisual geolocation, we propose a three-stage framework: (1) a Perception stage that utilizes a mixture-autoregressive sparse autoencoder to decompose noisy audio into semantically grounded "acoustic atoms"; (2) a Multimodal Reasoning stage that employs an MLLM finetuned via Group Relative Policy Optimization (GRPO) to synthesize these atoms with visual features; and (3) a Precision Prediction stage using Riemannian Flow Matching on the $S^2$ manifold. Our experiments demonstrate that our framework significantly outperforms unimodal baselines. These results entail that interpretable perception of the soundscape provides a critical, orthogonal signal that, when coupled with multimodal reasoning, enables high-precision global localization.

Related papers

• Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization [69.67186845167568]
  Open-vocabulary audio-visual event localization (OV-AVEL) jointly models audio-visual cues to recognize and temporally localize events.<n>Existing methods primarily learn joint audio-visual representations in Euclidean space.<n>We propose a hierarchical semantic constrained heterogeneous graph (HSCHG) for audio-visual event localization framework.
  arXiv  Detail & Related papers  (2026-06-05T08:23:58Z)
• ERGeoBench:A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models [17.86934121595618]
  ERGeoBench is a diagnostic benchmark for vision-driven embodied geo-localization.<n> benchmark contains 2,207 globally distributed street-view panoramas.<n> ERGeoBench provides a unified framework for diagnosing and advancing human-like embodied geo-localization.
  arXiv  Detail & Related papers  (2026-05-29T12:49:17Z)
• The Sonar Moment: Benchmarking Audio-Language Models in Audio Geo-Localization [31.970828788340782]
  We introduce AGL1K, the first audio geo-localization benchmark for audio language models (ALMs)<n>To extract reliably localizable samples from a crowd-sourced platform, we propose the Audio Localizability metric.<n>We find that closed-source models substantially outperform open-source models, and that linguistic clues often dominate as a scaffold for prediction.
  arXiv  Detail & Related papers  (2026-01-06T18:13:24Z)
• The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models [17.675850481660863]
  We introduce a hierarchical framework for Auditory Scene Analysis (ASA)<n> guided by this framework, we introduce a system that enables models like Qwen2-Audio to understand and reason about the complex acoustic world.<n>Our work provides a clear pathway for leveraging the powerful reasoning abilities of large models towards holistic acoustic scene analysis.
  arXiv  Detail & Related papers  (2026-01-06T11:54:47Z)
• SPUR: A Plug-and-Play Framework for Integrating Spatial Audio Understanding and Reasoning into Large Audio-Language Models [62.14165748145729]
  We introduce SPUR, a lightweight, plug-in approach that equips large audio-speaker models with spatial perception.<n>SPUR consists of: (i) a First-Order Ambisonics (FOA) encoder that maps channels to rotation-aware, listener-centric spatial features, integrated into target LALMs via a multimodal adapter; and (ii) SPUR-Set, a spatial QA dataset combining open-source FOA recordings with controlled simulations, emphasizing relative direction, elevation, distance, and overlap for supervised spatial reasoning.
  arXiv  Detail & Related papers  (2025-11-10T01:29:26Z)
• GLip: A Global-Local Integrated Progressive Framework for Robust Visual Speech Recognition [72.29071664964633]
  We propose GLip, a Global-Local Integrated Progressive framework designed for robust visual speech recognition (VSR)<n>GLip learns to align both global and local visual features with corresponding acoustic speech units using easily accessible audio-visual data.<n>In the second stage, we introduce a Contextual Enhancement Module (CEM) to dynamically integrate local features with relevant global context.
  arXiv  Detail & Related papers  (2025-09-19T14:36:01Z)
• Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling [50.8215545241128]
  We propose a.<n> Boundary Modeling Network (HBMNet), which includes three modules: an Audio-Visual Feature, a.<n> Coarse Proposal Generator and a Fine-Hierarchical Probabilities Generator.<n>From the modality perspective, we enhance audio-visual encoding and fusion, reinforced by frame-level supervision.<n>Experiments show that encoding and fusion primarily improve precision, while frame-level supervision recall.
  arXiv  Detail & Related papers  (2025-08-04T02:41:09Z)
• SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing [17.185628958975528]
  3D spatial reasoning in dynamic, audio-visual environments is a cornerstone of human cognition.<n> SAVVY is the first benchmark for 3D spatial reasoning in dynamic scenes with synchronized spatial audio.
  arXiv  Detail & Related papers  (2025-06-04T19:11:20Z)
• QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition [47.103732403296654]
  Multi-source semantic space can be represented as the Cartesian product of single-source sub-spaces. We introduce a global-to-local quantization mechanism, which distills knowledge from stable global (clip-level) features into local (frame-level) ones. Experiments demonstrate that our semantically decomposed audio representation significantly improves AVS performance.
  arXiv  Detail & Related papers  (2023-09-29T20:48:44Z)
• Hierarchical Local-Global Transformer for Temporal Sentence Grounding [58.247592985849124]
  This paper studies the multimedia problem of temporal sentence grounding. It aims to accurately determine the specific video segment in an untrimmed video according to a given sentence query.
  arXiv  Detail & Related papers  (2022-08-31T14:16:56Z)
• Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation [87.03299519917019]
  We propose a dual-scale graph transformer (DUET) for joint long-term action planning and fine-grained cross-modal understanding. We build a topological map on-the-fly to enable efficient exploration in global action space. The proposed approach, DUET, significantly outperforms state-of-the-art methods on goal-oriented vision-and-language navigation benchmarks.
  arXiv  Detail & Related papers  (2022-02-23T19:06:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.