Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification

• URL: http://arxiv.org/abs/2604.10695v2
• Date: Tue, 14 Apr 2026 03:39:04 GMT
• Title: Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification
• Abstract summary: R$2$ScP is a novel framework that shifts the paradigm of missing modality handling from traditional generative imputation to retrieval-based recovery.<n>Specifically, we leverage cross-modal retrieval via unified semantic embeddings to acquire missing domain-specific knowledge.
• Score: 30.75902088237621
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent Audio-Visual Question Answering (AVQA) methods have advanced significantly. However, most AVQA methods lack effective mechanisms for handling missing modalities, suffering from severe performance degradation in real-world scenarios with data interruptions. Furthermore, prevailing methods for handling missing modalities predominantly rely on generative imputation to synthesize missing features. While partially effective, these methods tend to capture inter-modal commonalities but struggle to acquire unique, modality-specific knowledge within the missing data, leading to hallucinations and compromised reasoning accuracy. To tackle these challenges, we propose R$^{2}$ScP, a novel framework that shifts the paradigm of missing modality handling from traditional generative imputation to retrieval-based recovery. Specifically, we leverage cross-modal retrieval via unified semantic embeddings to acquire missing domain-specific knowledge. To maximize semantic restoration, we introduce a context-aware adaptive purification mechanism that eliminates latent semantic noise within the retrieved data. Additionally, we employ a two-stage training strategy to explicitly model the semantic relationships between knowledge from different sources. Extensive experiments demonstrate that R$^{2}$ScP significantly improves AVQA and enhances robustness in modal-incomplete scenarios.

Related papers

• Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities [0.0]
  Large-scale multimodal models (LMMs) have achieved superior performance in visual recognition by synergizing information across diverse, massive-scale paired modalities.<n>In real-world scenarios, however, missing-modality inputs are ubiquitous, causing models optimized for modality-complete data to exhibit precipitous performance degradation.<n>Existing research has introduced prompt learning to mitigate this issue, typically by generating dynamic prompts from instance-level features.<n>As higher missing rates increase the proportion of incomplete modalities, the resulting instability in prompt learning limits the model's performance.
  arXiv  Detail & Related papers  (2026-06-29T17:36:47Z)
• Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration [40.720288165545476]
  We introduce an enhanced diffusion model as a pluggable mid-stage training module to effectively restore missing features.<n>Our strategy introduces two key innovations: (I) Dynamic Modality Gating, which adaptively leverages conditional features to steer the generation of semantically consistent features; (II) Cross-Modal Mutual Learning mechanism, which bridges the semantic spaces of dual encoders to achieve bidirectional alignment.
  arXiv  Detail & Related papers  (2026-02-03T06:06:35Z)
• Buffer replay enhances the robustness of multimodal learning under missing-modality [9.512378886218395]
  We introduce REplay Prompting (REP), which builds modality-wise buffers and replays them in deeper layers to mitigate information loss as network depth increases.<n>Experiments on vision-language, vision-language-audio, and temporal multimodal benchmarks demonstrate that REP consistently outperforms prior methods under both single- and multi-modality missing scenarios.<n>These results establish REP as a lightweight and effective paradigm for robust multimodal learning in challenging missing-modality environments.
  arXiv  Detail & Related papers  (2025-11-28T10:55:31Z)
• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering [54.72902502486611]
  ReAG is a Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages.<n>ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.
  arXiv  Detail & Related papers  (2025-11-27T19:01:02Z)
• Synergistic Prompting for Robust Visual Recognition with Missing Modalities [13.821274074204082]
  Large-scale multi-modal models have demonstrated remarkable performance across various visual recognition tasks.<n>The presence of missing or incomplete modality inputs often leads to significant performance degradation.<n>We propose a novel Synergistic Prompting framework for robust visual recognition with missing modalities.
  arXiv  Detail & Related papers  (2025-07-10T14:28:12Z)
• ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion [25.1725138364452]
  We introduce an Attention-based Diffusion model for Missing Modalities feature Completion (ADMC)<n>Our framework independently trains feature extraction networks for each modality, preserving their unique characteristics and avoiding over-coupling.<n>Our approach achieves state-of-the-art results on the IEMOCAP and MIntRec benchmarks, demonstrating its effectiveness in both missing and complete modality scenarios.
  arXiv  Detail & Related papers  (2025-07-08T03:08:52Z)
• How Far Are We from Generating Missing Modalities with Foundation Models? [49.425856207329524]
  We propose an agentic framework tailored for missing modality reconstruction.<n>Our method reduces FID for missing image reconstruction by at least 14% and MER for missing text reconstruction by at least 10% compared to baselines.
  arXiv  Detail & Related papers  (2025-06-04T03:22:44Z)
• DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery [71.6345505427213]
  DPMesh is an innovative framework for occluded human mesh recovery. It capitalizes on the profound diffusion prior about object structure and spatial relationships embedded in a pre-trained text-to-image diffusion model.
  arXiv  Detail & Related papers  (2024-04-01T18:59:13Z)
• A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition [53.800937914403654]
  Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. We propose a novel Multimodal Distribution Approximation with Knowledge Distillation (MDA-KD) framework to reduce over-reliance on the audio modality.
  arXiv  Detail & Related papers  (2024-03-07T06:06:55Z)
• Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities [76.08541852988536]
  We propose to use invariant features for a missing modality imagination network (IF-MMIN) We show that the proposed model outperforms all baselines and invariantly improves the overall emotion recognition performance under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2022-10-27T12:16:25Z)
• Self-attention fusion for audiovisual emotion recognition with incomplete data [103.70855797025689]
  We consider the problem of multimodal data analysis with a use case of audiovisual emotion recognition. We propose an architecture capable of learning from raw data and describe three variants of it with distinct modality fusion mechanisms.
  arXiv  Detail & Related papers  (2022-01-26T18:04:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.