Head-Pose-Aware Visual Speech Recognition with FiLM Modulation

• URL: http://arxiv.org/abs/2606.00751v1
• Date: Sat, 30 May 2026 14:35:47 GMT
• Title: Head-Pose-Aware Visual Speech Recognition with FiLM Modulation
• Abstract summary: We propose a pose-aware phoneme-level framework, termed HP-VSR-ResFiLM, that explicitly incorporates head-pose information into visual feature extraction.<n>We show that HP-VSR-ResFiLM achieves competitive performance under comparable training conditions, word error rates (WER) of 25.0% and 33.2%, respectively, without relying on additional training data.
• Score: 1.3686940157259604
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual Speech Recognition (VSR) aims to recognize speech from visual cues such as lip movements, but its performance is fundamentally limited by viseme ambiguity and pose-induced variations that introduce geometric distortions and occlusions. Existing approaches mainly rely on linguistic context or implicit invariance, leaving visual representations insufficiently robust under non-frontal views. In this work, we propose a pose-aware phoneme-level framework, termed HP-VSR-ResFiLM, that explicitly incorporates head-pose information into visual feature extraction. The proposed framework adopts a two-stage pipeline consisting of a pose-conditioned visual encoder in Stage 1 and a pretrained NLLB language model in Stage 2 for phoneme-to-text reconstruction. Specifically, Stage 1 incorporates a pose-conditioned residual Feature-wise Linear Modulation (FiLM) block after the 2D CNN frontend to adaptively refine visual representations using head-pose information. Experiments on LRS2 and LRS3 demonstrate that HP-VSR-ResFiLM achieves competitive performance under comparable training conditions, attaining word error rates (WER) of 25.0% and 33.2%, respectively, without relying on additional training data. Ablation studies further show that a single residual FiLM block consistently improves overall WER, while deeper modulation at Layers 3 and 4 provides larger gains for samples with yaw angles greater than 30° without degrading performance for smaller pose variations. These findings demonstrate that explicit pose-aware feature modulation offers an effective and computationally efficient solution for improving VSR robustness in unconstrained settings.

Related papers

• Wiener Representation Filtering for VLM Hallucination Suppression [45.33665443468439]
  We propose a training-free, post-hoc representation editing technique that operates in the representation space of the language backbone.<n>By modeling hidden states as a superposition of truthful and hallucination filtering components, we derive a Wiener-type estimator.<n>Experiments on LLaVA1.53, MiniGPT-4, Gemma POPE, and mOwl2 demonstrate consistent reductions in object hallucination.
  arXiv  Detail & Related papers  (2026-08-08T14:51:17Z)
• ExACT: Exemplar-Driven Calibrated Refinement for Training-Free Visual Grounding in Remote Sensing Images [51.53960096934913]
  Remote sensing visual grounding (RSVG) aims to locate specific objects in high-resolution RS imagery using free-form natural language descriptions.<n>Recent advances in multimodal large language models (MLLMs) show great potential for such open-vocabulary RSVG.<n>We propose Exemplar-driven Calibrated Refinement (ExACT), a training-free framework driven by a one-shot visual prompting mechanism.
  arXiv  Detail & Related papers  (2026-06-27T13:50:39Z)
• Diffusion Large Language Models for Visual Speech Recognition [56.81307584718608]
  We propose the first Diffusion Large Language Model (DLLM)-based Visual Speech Recognition (VSR) framework.<n>DLLM-VSR commits high-confidence positions early and uses the committed tokens as bidirectional context to refine ambiguous ones.<n>We develop length-guided candidate decoding, which uses video duration to construct plausible transcript-length hypotheses.<n>The proposed method achieves a state-of-the-art WER of 19.5% on LRS3 using only its labeled training data.
  arXiv  Detail & Related papers  (2026-05-27T13:22:08Z)
• A Two-Stage Dual-Modality Model for Facial Emotional Expression Recognition [5.014305646574725]
  This paper addresses the expression (EXPR) recognition challenge in the 10th Affective Behavior Analysis in-the-Wild (ABAW) workshop and competition.<n>It requires frame-level classification of eight facial emotional expressions from unconstrained videos.<n>We propose a two-stage dual-modal (audio-visual) model to address these difficulties.
  arXiv  Detail & Related papers  (2026-03-12T17:45:12Z)
• Suppressing VLM Hallucinations with Spectral Representation Filtering [49.52703800684483]
  Vision-language models (VLMs) frequently produce hallucinations in the form of descriptions of objects, attributes, or relations that do not exist in the image.<n>We introduce Spectral Representation Filtering (SRF), a lightweight, training-free method to suppress such hallucinations by analyzing and correcting the covariance structure of the model's representations.
  arXiv  Detail & Related papers  (2025-11-15T13:49:27Z)
• GLip: A Global-Local Integrated Progressive Framework for Robust Visual Speech Recognition [72.29071664964633]
  We propose GLip, a Global-Local Integrated Progressive framework designed for robust visual speech recognition (VSR)<n>GLip learns to align both global and local visual features with corresponding acoustic speech units using easily accessible audio-visual data.<n>In the second stage, we introduce a Contextual Enhancement Module (CEM) to dynamically integrate local features with relevant global context.
  arXiv  Detail & Related papers  (2025-09-19T14:36:01Z)
• Phoneme-Level Visual Speech Recognition via Point-Visual Fusion and Language Model Reconstruction [1.778037147204838]
  Visual Automatic Speech Recognition (V-ASR) is a challenging task that involves interpreting spoken language solely from visual information, such as lip movements and facial expressions.<n>Existing methods often aim to predict words directly from visual cues, but they commonly suffer from high error rates due to viseme ambiguity.<n>We propose a novel phoneme-based two-stage framework that fuses visual and landmark motion features.
  arXiv  Detail & Related papers  (2025-07-25T00:38:39Z)
• WSCLoc: Weakly-Supervised Sparse-View Camera Relocalization [42.85368902409545]
  WSCLoc is a system capable of being customized to various deep learning-based relocalization models. In the initial stage, WSCLoc employs a multilayer perceptron-based structure called WFT-NeRF to co-optimize image reconstruction quality. In the second stage, we co-optimize the pre-trained WFT-NeRF and WFT-Pose.
  arXiv  Detail & Related papers  (2024-03-22T15:15:44Z)
• OV-NeRF: Open-vocabulary Neural Radiance Fields with Vision and Language Foundation Models for 3D Semantic Understanding [9.25233177676278]
  OV-NeRF exploits potential of pre-trained vision and language foundation models to enhance semantic field learning. Our approach achieves a significant improvement of 20.31% and 18.42% in mIoU metric on Replica and ScanNet, respectively.
  arXiv  Detail & Related papers  (2024-02-07T08:19:57Z)
• Lip2Vec: Efficient and Robust Visual Speech Recognition via Latent-to-Latent Visual to Audio Representation Mapping [4.271091833712731]
  We propose a simple approach, named Lip2Vec that is based on learning a prior model. The proposed model compares favorably with fully-supervised learning methods on the LRS3 dataset achieving 26 WER. We believe that reprogramming the VSR as an ASR task narrows the performance gap between the two and paves the way for more flexible formulations of lip reading.
  arXiv  Detail & Related papers  (2023-08-11T12:59:02Z)
• Learning to Decompose Visual Features with Latent Textual Prompts [140.2117637223449]
  We propose Decomposed Feature Prompting (DeFo) to improve vision-language models. Our empirical study shows DeFo's significance in improving the vision-language models.
  arXiv  Detail & Related papers  (2022-10-09T15:40:13Z)
• End-to-end Audio-visual Speech Recognition with Conformers [65.30276363777514]
  We present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer) In particular, the audio and visual encoders learn to extract features directly from raw pixels and audio waveforms. We show that our proposed models raise the state-of-the-art performance by a large margin in audio-only, visual-only, and audio-visual experiments.
  arXiv  Detail & Related papers  (2021-02-12T18:00:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.