AMAVA: Adaptive Motion-Aware Video-to-Audio Framework for Visually-Impaired Assistance

• URL: http://arxiv.org/abs/2604.23909v1
• Date: Sun, 26 Apr 2026 23:03:15 GMT
• Title: AMAVA: Adaptive Motion-Aware Video-to-Audio Framework for Visually-Impaired Assistance
• Abstract summary: We present AMAVA, a novel real-time video-to-audio framework that converts mobile device video into contextually relevant sound effects or text-to-speech descriptions.<n>In static environments, AMAVA generates spoken audio scene descriptions for situational awareness.<n>In high-movement situations, it prioritizes safety by delivering sound cues, such as spoken hazard alerts and environmental sound effects.
• Score: 1.8199179205103146
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Navigational aids for blind and low vision individuals struggle conveying dynamic real-world environments, leading to cognitive overload from continuous, undifferentiated feedback. We present AMAVA, a novel real-time video-to-audio framework that converts mobile device video into contextually relevant sound effects or text-to-speech descriptions. We propose a motion-aware pipeline using a lightweight AI classification model to distinguish between low and high-movement scenes followed by a real-time text-to-audio synthesis pipeline to enhance environmental perception more efficiently. In static environments, AMAVA generates spoken audio scene descriptions for situational awareness. In high-movement situations, it prioritizes safety by delivering sound cues, such as spoken hazard alerts and environmental sound effects. These audio outputs are produced by a decoder-only transformer-based vision-language model with mixture-of-experts and cross-modal attention for visual understanding, in conjunction with neural text-to-speech and natural sound synthesis networks. The proposed framework uses prompt-based caching and category-specific throttling to avoid auditory clutter and minimize latency. We present a comprehensive evaluation of the system, including a real-time navigation study comparing a white cane alone versus with AMAVA, that shows a significant increase in user confidence and perceived safety.

Related papers

• Hijacking Large Audio-Language Models via Context-Agnostic and Imperceptible Auditory Prompt Injection [22.306688903148046]
  Large audio-language models (LALMs) power intelligent voice interactions by tightly integrating audio and text.<n>We propose textitAudioHijack, a framework that generates context-agnostic and imperceptible adversarial audio to hijack LALMs.<n>Experiments on 13 state-of-the-art LALMs show consistent hijacking across 6 categories misbehavior.
  arXiv  Detail & Related papers  (2026-04-16T04:22:11Z)
• 3rd Place of MeViS-Audio Track of the 5th PVUW: VIRST-Audio [6.447274127678917]
  We present VIRST-Audio, a framework built upon a pretrainedOS model integrated with a vision-language architecture.<n>We convert input audio into text using an ASR module and perform segmentation using text-based supervision.<n>We evaluate our approach on the MeViS-Audio track of the 5th PVUW Challenge, where VIRST-Audio achieves 3rd place.
  arXiv  Detail & Related papers  (2026-03-24T12:23:10Z)
• Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation [26.766367856312694]
  We formalize Vision-Sound-Language-Action (VSLA) as a continuous control paradigm conditioned on vision, streaming audio, language, and proprioception under delayed decision loops.<n>As an instantiation, we introduce HEAR, a VSLA framework integrating four components: (i) a streaming Historizer to maintain a compact, causal audio context across execution gaps; (ii) an Envisioner adapted from omni foundation models to reason over multi-sensory inputs; (iii) an Advancer, formulated as an audio world model, to learn temporal dynamics by predicting near-future audio codes; and (iv) a flow
  arXiv  Detail & Related papers  (2026-03-17T03:22:30Z)
• Semantics-Aware Human Motion Generation from Audio Instructions [25.565742045932236]
  This paper explores a new task, where audio signals are used as conditioning inputs to generate motions that align with the semantics of the audio.<n>We propose an end-to-end framework using a masked generative transformer, enhanced by a memory-retrieval attention module to handle sparse and lengthy audio inputs.
  arXiv  Detail & Related papers  (2025-05-29T14:16:27Z)
• Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioning [37.17910848101769]
  Current vision-guided audio captioning systems fail to address audiovisual misalignment in real-world scenarios.<n>We present an entropy-aware gated fusion framework that dynamically modulates visual information flow through cross-modal uncertainty quantification.<n>We also develop a batch-wise audiovisual shuffling technique that generates synthetic mismatched training pairs.
  arXiv  Detail & Related papers  (2025-05-28T07:08:17Z)
• Exploiting Temporal Audio-Visual Correlation Embedding for Audio-Driven One-Shot Talking Head Animation [62.218932509432314]
  Inherently, the temporal relationship of adjacent audio clips is highly correlated with that of the corresponding adjacent video frames.<n>We learn audio-visual correlations and integrate the correlations to help enhance feature representation and regularize final generation.
  arXiv  Detail & Related papers  (2025-04-08T07:23:28Z)
• AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection [57.649223695021114]
  We present a novel weakly supervised framework that leverages audio-visual collaboration for robust video anomaly detection.<n>Our framework demonstrates superior performance across multiple benchmarks, with audio integration significantly boosting anomaly detection accuracy.
  arXiv  Detail & Related papers  (2025-04-06T13:59:16Z)
• Sonic: Shifting Focus to Global Audio Perception in Portrait Animation [43.63279351897198]
  The study of talking face generation mainly explores the intricacies of synchronizing facial movements and crafting visually appealing, temporally-coherent animations.<n>We propose a novel paradigm, dubbed as Sonic, to leverage global audio knowledge and enhance overall perception.<n>Extensive experiments demonstrate that the novel audio-driven paradigm outperform existing SOTA methodologies in terms of video quality, temporally consistency, lip synchronization precision, and motion diversity.
  arXiv  Detail & Related papers  (2024-11-25T12:24:52Z)
• Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial Cues [80.53407593586411]
  We focus on leveraging facial cues beyond the lip region for robust Audio-Visual Speech Enhancement (AVSE) We propose a Dual Attention Cooperative Framework, DualAVSE, to ignore speech-unrelated information, capture speech-related information with facial cues, and dynamically integrate it with the audio signal for AVSE.
  arXiv  Detail & Related papers  (2023-11-24T04:30:31Z)
• AdVerb: Visually Guided Audio Dereverberation [49.958724234969445]
  We present AdVerb, a novel audio-visual dereverberation framework. It uses visual cues in addition to the reverberant sound to estimate clean audio.
  arXiv  Detail & Related papers  (2023-08-23T18:20:59Z)
• Motion and Context-Aware Audio-Visual Conditioned Video Prediction [58.9467115916639]
  We decouple the audio-visual conditioned video prediction into motion and appearance modeling. The multimodal motion estimation predicts future optical flow based on the audio-motion correlation. We propose context-aware refinement to address the diminishing of the global appearance context.
  arXiv  Detail & Related papers  (2022-12-09T05:57:46Z)
• Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis [67.73554826428762]
  We propose a novel audio-visual speech enhancement framework for high-fidelity telecommunications in AR/VR. Our approach leverages audio-visual speech cues to generate the codes of a neural speech, enabling efficient synthesis of clean, realistic speech from noisy signals.
  arXiv  Detail & Related papers  (2022-03-31T17:57:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.