Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognition

• URL: http://arxiv.org/abs/2608.26213v1
• Date: Wed, 26 Aug 2026 08:25:15 GMT
• Title: Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognition
• Abstract summary: Contrastive decoding (CD) originally introduced to stabilize large language model (LLM)-based audio-visual speech recognition (AVSR) systems.<n>We propose reliability-aware scaling of CD for AVSR.
• Score: 50.15288513086986
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model (LLM)-based audio-visual speech recognition (AVSR) systems are robust under noise. Contrastive decoding (CD), originally introduced to stabilize LLM generation by contrasting a weaker model against a stronger one at inference time, adjusts predictions without additional training. In this work, we apply CD to AVSR by contrasting audio-only conditioning with full audio-visual conditioning within the same underlying model. However, using a fixed contrastive strength introduces a trade-off across noise levels: stronger intervention helps under severe noise but may over-correct reliable predictions in clean conditions. We propose reliability-aware scaling of CD for AVSR. Instead of using a fixed strength, we adaptively modulate the contrastive influence at each token based on reliability signals derived from attention dynamics and inter-model predictive divergence. Experiments on LRS3 show consistent improvements across clean and low-SNR conditions.

Related papers

• Equilibrium Forcing: Adaptive Video Generation Without Noise Conditioning [85.93126001283169]
  We introduce Equilibrium Forcing (EqF), a framework for video denoising generative models without noise level conditioning.<n>This flexibility allows for inference-time algorithms that operate in a closed loop by adapting to feedback from the sample.
  arXiv  Detail & Related papers  (2026-08-11T00:02:32Z)
• SATB-VR: Training Few-Step Video Restoration Diffusion Model using SNR-Aware Trajectory Blending [55.764018145256216]
  We present SATB-VR, a few-step paradigm that jump-starts the denoising process via an auxiliary predictor.<n>We show that SATB-VR performs favorably against existing approaches on synthetic, real-world, and AIGC benchmarks.
  arXiv  Detail & Related papers  (2026-06-27T01:32:16Z)
• Test-Time Self-Adaptive Conditioning for Stable Audio-Driven Talking-Head Generation [28.732587811107777]
  We introduce Test-Time Self-Adaptive Conditioning (TT-SAC), a parameter-free inference framework that enables pretrained talking-head generators to adapt their conditioning representations during inference without retraining, gradient updates, or additional supervision.<n>A single adaptation step approximates a self-consistent equilibrium of the generative process, stabilizing identity and motion across time.<n>Experiments on state-of-the-art talking-head generators and benchmark datasets demonstrate consistent improvements in lip-sync accuracy, temporal coherence, identity preservation, and perceptual fidelity.
  arXiv  Detail & Related papers  (2026-05-25T06:45:29Z)
• Adaptive Guidance for Retrieval-Augmented Masked Diffusion Models [60.25003813232697]
  We propose Adaptive Retrieval-Augmented Masked Diffusion (ARAM) as a training-free adaptive guidance framework for Masked Diffusion Models (MDMs)<n>ARAM calibrates the guidance scale during denoising according to the Signal-to-Noise Ratio (SNR) of the distributional shift induced by retrieved context.<n>Experiments on multiple knowledge-intensive QA benchmarks show that ARAM improves overall QA performance over competitive RAG baselines.
  arXiv  Detail & Related papers  (2026-03-18T12:54:50Z)
• Boosting ASR Robustness via Test-Time Reinforcement Learning with Audio-Text Semantic Rewards [8.109014000578766]
  We present ASR-TRA, a novel Testtime Reinforcement Adaptation framework inspired by causal intervention.<n>Our method achieves higher accuracy while maintaining lower latency than existing TTA baselines.<n>Our approach provides a practical and robust solution for deploying ASR systems in challenging real-world conditions.
  arXiv  Detail & Related papers  (2026-03-05T14:43:15Z)
• NAACL: Noise-AwAre Verbal Confidence Calibration for LLMs in RAG Systems [53.52419750390942]
  Large language models (LLMs) are used in mission-critical factual domains.<n>LLMs exhibit poor calibration performance due to noisy retrieved contexts.<n>We propose NAACL Rules (Noise-AwAre Confidence CaLibration Rules) to provide a principled foundation for resolving overconfidence under noise.
  arXiv  Detail & Related papers  (2026-01-16T05:38:25Z)
• Test-time Adaptive Hierarchical Co-enhanced Denoising Network for Reliable Multimodal Classification [55.56234913868664]
  We propose Test-time Adaptive Hierarchical Co-enhanced Denoising Network (TAHCD) for reliable learning on multimodal data.<n>The proposed method achieves superior classification performance, robustness, and generalization compared with state-of-the-art reliable multimodal learning approaches.
  arXiv  Detail & Related papers  (2026-01-12T03:14:12Z)
• Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusion [46.072071890391356]
  We propose a novel framework that adaptively reweights audio and visual features based on token-level acoustic corruption scores.<n>Using an audio-visual feature fusion-based router, our method down-weights unreliable audio tokens and reinforces visual cues through cross-attention gated in each decoder layer.<n> Experiments on LRS3 demonstrate that our approach achieves an 16.51-42.67% relative reduction in word error rate compared to AV-HuBERT.
  arXiv  Detail & Related papers  (2025-08-26T07:05:48Z)
• Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Models [1.0579965347526206]
  Large language models (LLMs) often produce inaccurate or misleading content-hallucinations.<n>Noise-Augmented Fine-Tuning (NoiseFiT) is a novel framework that leverages adaptive noise injection to enhance model robustness.<n>NoiseFiT selectively perturbs layers identified as either high-SNR (more robust) or low-SNR (potentially under-regularized) using a dynamically scaled Gaussian noise.
  arXiv  Detail & Related papers  (2025-04-04T09:27:19Z)
• Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder [30.318947721658862]
  We propose to include noise information in the training phase by using a noise-aware encoder trained on noisy-clean speech pairs. We show that our proposed noise-aware VAE outperforms the standard VAE in terms of overall distortion without increasing the number of model parameters.
  arXiv  Detail & Related papers  (2021-02-17T11:40:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.