The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?

• URL: http://arxiv.org/abs/2606.07435v2
• Date: Mon, 08 Jun 2026 10:07:12 GMT
• Title: The Lipreading Gap: Do VSR Models Perceive Visual Speech Like Human Lipreaders?
• Abstract summary: Visual speech recognition models now surpass human lipreaders on benchmarks, but do such gains establish human-like visual speech perception?<n>We compare three VSR systems with human baselines on the MaFI word-level lipreading dataset using word, character, phoneme, and viseme-level metrics.<n>Our work demonstrates that VSR systems rely primarily on language cues from training data rather than visual perception, failing to bind visual features into meaningful words.
• Score: 14.788453263873444
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual speech recognition (VSR) models now surpass human lipreaders on benchmarks, but do such gains establish human-like visual speech perception? To explore this, we compare three VSR systems with human baselines on the MaFI word-level lipreading dataset using word, character, phoneme, and viseme-level metrics. Although models achieve higher overall accuracy, they succeed and fail on different words than humans. A text-only n-gram baseline given only a few initial phonemes rivals human lipreading. VSR word-level errors are consistently better explained by training word frequency than by the visual informativeness of words. Viseme accuracies, confusion matrices and human-model correlations further show that models gain most on visemes humans find hardest, and show much weaker dependence on visual clarity. Our work demonstrates that VSR systems rely primarily on language cues from training data rather than visual perception, failing to bind visual features into meaningful words.

Related papers

• See the Text: From Tokenization to Visual Reading [63.10220471118435]
  SeeTok renders text as images (visual-text) and leverages pretrained multimodal computations to interpret them.<n>Across three different language tasks, SeeTok matches or surpasses subword tokenizers while requiring 4.43 times fewer tokens and reducing FLOPs by 70.5%.<n>SeeTok signals a shift from symbolic tokenization to human-like visual reading, and takes a step toward more natural and cognitively inspired language models.
  arXiv  Detail & Related papers  (2025-10-21T17:34:48Z)
• The Visual Iconicity Challenge: Evaluating Vision-Language Models on Sign Language Form-Meaning Mapping [1.5767445615203355]
  Visual Iconicity Challenge adapts psycholinguistic measures to evaluate vision-language models.<n>We assess 13 state-of-the-art VLMs in zero- and few-shot settings on Sign Language of the Netherlands.<n>Models with stronger phonological form prediction correlate better with human iconicity judgment.
  arXiv  Detail & Related papers  (2025-10-09T17:21:59Z)
• VALLR: Visual ASR Language Model for Lip Reading [28.561566996686484]
  Lip Reading, or Visual Automatic Speech Recognition, is a complex task requiring the interpretation of spoken language exclusively from visual cues.<n>We propose a novel two-stage, phoneme-centric framework for Visual Automatic Speech Recognition (V-ASR)<n>First, our model predicts a compact sequence of phonemes from visual inputs using a Video Transformer with a CTC head.<n>This phoneme output then serves as the input to a fine-tuned Large Language Model (LLM), which reconstructs coherent words and sentences.
  arXiv  Detail & Related papers  (2025-03-27T11:52:08Z)
• Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation [55.15299351110525]
  This paper explores sentence-level multilingual Visual Speech Recognition (VSR) that can recognize different languages with a single trained model. We propose a novel training strategy, processing with visual speech units. We set new state-of-the-art multilingual VSR performances by achieving comparable performances to the previous language-specific VSR models.
  arXiv  Detail & Related papers  (2024-01-18T08:46:02Z)
• Visual Grounding Helps Learn Word Meanings in Low-Data Regimes [47.7950860342515]
  Modern neural language models (LMs) are powerful tools for modeling human sentence production and comprehension. But to achieve these results, LMs must be trained in distinctly un-human-like ways. Do models trained more naturalistically -- with grounded supervision -- exhibit more humanlike language learning? We investigate this question in the context of word learning, a key sub-task in language acquisition.
  arXiv  Detail & Related papers  (2023-10-20T03:33:36Z)
• Jointly Learning Visual and Auditory Speech Representations from Raw Data [108.68531445641769]
  RAVEn is a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our design is asymmetric w.r.t. driven by the inherent differences between video and audio. RAVEn surpasses all self-supervised methods on visual speech recognition.
  arXiv  Detail & Related papers  (2022-12-12T21:04:06Z)
• Sub-word Level Lip Reading With Visual Attention [88.89348882036512]
  We focus on the unique challenges encountered in lip reading and propose tailored solutions. We obtain state-of-the-art results on the challenging LRS2 and LRS3 benchmarks when training on public datasets. Our best model achieves 22.6% word error rate on the LRS2 dataset, a performance unprecedented for lip reading models.
  arXiv  Detail & Related papers  (2021-10-14T17:59:57Z)
• Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition [90.61063126619182]
  We evaluate the effects of different facial regions with state-of-the-art visual speech recognition models. We find that incorporating information from extraoral facial regions, even the upper face, consistently benefits VSR performance.
  arXiv  Detail & Related papers  (2020-03-06T13:52:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.