NeuroLip: An Event-driven Spatiotemporal Learning Framework for Cross-Scene Lip-Motion-based Visual Speaker Recognition

• URL: http://arxiv.org/abs/2604.15718v1
• Date: Fri, 17 Apr 2026 05:42:17 GMT
• Title: NeuroLip: An Event-driven Spatiotemporal Learning Framework for Cross-Scene Lip-Motion-based Visual Speaker Recognition
• Abstract summary: Lip motion encodes subject-specific behavioral dynamics driven by consistent articulation patterns and muscle coordination.<n>NeuroLip is an event-based framework that captures fine-grained lip dynamics under a strict yet practical cross-scene protocol.<n>NeuroLip achieves near-perfect matched-scene accuracy and robust cross-scene recognition, attaining over 71% accuracy on unseen viewpoints.
• Score: 17.96933456290729
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Visual speaker recognition based on lip motion offers a silent, hands-free, and behavior-driven biometric solution that remains effective even when acoustic cues are unavailable. Compared to traditional methods that rely heavily on appearance-dependent representations, lip motion encodes subject-specific behavioral dynamics driven by consistent articulation patterns and muscle coordination, offering inherent stability across environmental changes. However, capturing these robust, fine-grained dynamics is challenging for conventional frame-based cameras due to motion blur and low dynamic range. To exploit the intrinsic stability of lip motion and address these sensing limitations, we propose NeuroLip, an event-based framework that captures fine-grained lip dynamics under a strict yet practical cross-scene protocol: training is performed under a single controlled condition, while recognition must generalize to unseen viewing and lighting conditions. NeuroLip features a 1) Temporal-aware Voxel Encoding module with adaptive event weighting, 2) Structure-aware Spatial Enhancer that amplifies discriminative behavioral patterns by suppressing noise while preserving vertically structured motion information, and 3) Polarity Consistency Regularization mechanism to retain motion-direction cues encoded in event polarities. To facilitate systematic evaluation, we introduce DVSpeaker, a comprehensive event-based lip-motion dataset comprising 50 subjects recorded under four distinct viewpoint and illumination scenarios. Extensive experiments demonstrate that NeuroLip achieves near-perfect matched-scene accuracy and robust cross-scene generalization, attaining over 71% accuracy on unseen viewpoints and nearly 76% under low-light conditions, outperforming representative existing methods by at least 8.54%. The dataset and code are publicly available at https://github.com/JiuZeongit/NeuroLip.

Related papers

• Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding [63.599342750566485]
  SPAR is a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation.<n>Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively.
  arXiv  Detail & Related papers  (2026-08-29T09:58:24Z)
• Implicit Neural Speckle Denoising [8.506200151475559]
  We introduce a training-free framework that combines a neural representation with an aperture-aware maximum-likelihood to recover dynamic, speckle-free imagery directly from noisy observations.<n>Results show improved spatial fidelity, temporal consistency, and varying speckle statistics relative to classical, unsupervised, and supervised observations.
  arXiv  Detail & Related papers  (2026-08-06T20:36:51Z)
• SubtleTalk: Generating Controllable Weakly-correlated Facial Dynamics for 3D Talking Heads via Residual Flow Matching [21.891013914273714]
  Audio-driven 3D facial animation aims to synthesize realistic and temporally coherent motions from speech.<n>We present SubtleTalk, a framework for generating natural and controllable weakly correlated facial dynamics.<n>Our method significantly improves the realism and diversity of weakly correlated facial dynamics while preserving accurate lip synchronization.
  arXiv  Detail & Related papers  (2026-08-03T11:28:19Z)
• Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigation [59.80828456543202]
  We propose Token-Level Visual-Sensitivity Steering (TLVS) for hallucination mitigation.<n>This mechanism requires only minimal training for calibration and can be applied across diverse vision-language models.<n>We evaluate TLVS on several benchmarks, including POPE, AMBER, CHAIR (COCO), MMHal, and HallusionBench, demonstrating consistent improvements over previous steering methods.
  arXiv  Detail & Related papers  (2026-06-02T08:23:28Z)
• EventGait: Towards Robust Gait Recognition with Event Streams [50.890621860023]
  Event cameras offer microsecond temporal resolution and high dynamic range, naturally capturing robust dynamic cues and suppressing static noise.<n>We propose textbfEventGait, an end-to-end dual-stream framework that separately models motion and shape while preserving the advantages of events.<n>Our approach sets a new state of the art on both synthesized and real-world event-based gait benchmarks, highlighting the robustness and potential of event-driven gait analysis.
  arXiv  Detail & Related papers  (2026-05-21T08:12:04Z)
• Neural Visual Decoding via Cognitive guided Adaptive Blurring and Information Constrained Alignment [15.968559815427193]
  EEG-based visual decoding aims to establish a mapping between neural signals and visual semantics.<n>Existing approaches typically treat static visual features, ignoring the dynamic selectivity of human vision and the frequency specificity of neural oscillations.<n>We propose CAIA, a Cognitive-guided Adaptive blurring with Information-Constrained Alignment framework for Neural-Visual decoding.
  arXiv  Detail & Related papers  (2026-05-14T08:50:29Z)
• V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising [65.5867130156805]
  We present V-Co, a systematic study of visual co-denoising in a unified JiT-based framework.<n>Our study reveals four key ingredients for effective visual co-denoising.<n>V-Co outperforms the underlying pixel-space diffusion baseline and strong prior pixel-diffusion methods.
  arXiv  Detail & Related papers  (2026-03-17T17:01:54Z)
• MotionCFG: Boosting Motion Dynamics via Stochastic Concept Perturbation [60.48914865049489]
  MotionCFG is a framework that enhances motion dynamics by contrasting a target concept with its noise-blured counterparts.<n>Unlike explicit negations, this approach facilitates implicit hard negative guidance without shifting the global semantic identity.<n> MotionCFG consistently improves motion dynamics across state-of-the-art T2V frameworks.
  arXiv  Detail & Related papers  (2026-03-14T18:39:37Z)
• Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering [33.3588824921581]
  We introduce Vision-Language Introspection (VLI), a training-free inference framework that simulates a metacognitive self-correction process.<n>VLI achieves state-of-the-art performance on advanced models, reducing object hallucination rates by 12.67% on MMHal-Bench and improving accuracy by 5.8% on POPE.
  arXiv  Detail & Related papers  (2026-01-08T17:49:13Z)
• MotionTeller: Multi-modal Integration of Wearable Time-Series with LLMs for Health and Behavioral Understanding [4.158479111055355]
  MotionTeller is a generative framework that integrates minute-level wearable activity data with large language models (LLMs)<n>We construct a novel dataset of 54383 (actigraphy, text) pairs derived from real-world NHANES recordings, and train the model using cross-entropy loss with supervision only on the language tokens.<n>MotionTeller achieves high semantic fidelity (BERT-F1 = 0.924) and lexical accuracy (ROUGE-1 = 0.722), outperforming prompt-based baselines by 7 percent in ROUGE-1.
  arXiv  Detail & Related papers  (2025-12-25T04:37:07Z)
• DynaPURLS: Dynamic Refinement of Part-aware Representations for Skeleton-based Zero-Shot Action Recognition [51.80782323686666]
  We introduce textbfDynaPURLS, a unified framework that establishes robust, multi-scale visual-semantic correspondences.<n>Our framework leverages a large language model to generate hierarchical textual descriptions that encompass both global movements and local body-part dynamics.<n>Experiments on three large-scale benchmark datasets, including NTU RGB+D 60/120 and PKU-MMD, demonstrate that DynaPURLS significantly outperforms prior art.
  arXiv  Detail & Related papers  (2025-12-12T10:39:10Z)
• Context-aware Fairness Evaluation and Mitigation in LLMs [16.594400974742523]
  Large language models often display undesirable behaviors embedded in their internal representations.<n>Pruning-based methods provide a flexible and transparent way to reduce bias by adjusting the neurons responsible for certain behaviors.<n>We propose a dynamic, reversible, pruning-based framework that detects context-aware neuron activations and applies adaptive masking to modulate their influence during generation.
  arXiv  Detail & Related papers  (2025-10-21T06:13:17Z)
• An Exact Gradient Framework for Training Spiking Neural Networks [0.7366405857677227]
  Spiking neural networks inherently rely on the precise timing of discrete spike events for information processing.<n>We propose an event-driven learning framework that computes exact loss gradients with respect to synaptic weights and transmission delays.<n>Experiments on multiple benchmarks demonstrate significant gains in accuracy (up to 7%), timing precision, and robustness compared to existing methods.
  arXiv  Detail & Related papers  (2025-07-08T11:55:27Z)
• ReCoM: Realistic Co-Speech Motion Generation with Recurrent Embedded Transformer [58.49950218437718]
  We present ReCoM, an efficient framework for generating high-fidelity and generalizable human body motions synchronized with speech.<n>The core innovation lies in the Recurrent Embedded Transformer (RET), which integrates Dynamic Embedding Regularization (DER) into a Vision Transformer (ViT) core architecture.<n>To enhance model robustness, we incorporate the proposed DER strategy, which equips the model with dual capabilities of noise resistance and cross-domain generalization.
  arXiv  Detail & Related papers  (2025-03-27T16:39:40Z)
• Allostatic Control of Persistent States in Spiking Neural Networks for perception and computation [79.16635054977068]
  We introduce a novel model for updating perceptual beliefs about the environment by extending the concept of Allostasis to the control of internal representations.<n>In this paper, we focus on an application in numerical cognition, where a bump of activity in an attractor network is used as a spatial numerical representation.
  arXiv  Detail & Related papers  (2025-03-20T12:28:08Z)
• DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering [69.9557427451339]
  We propose a framework based on neural radiance field to pursue high-fidelity talking head generation. Specifically, neural radiance field takes lip movements features and personalized attributes as two disentangled conditions. We show that our method achieves significantly better results than state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-01-03T18:23:38Z)
• Neuromorphic Camera Denoising using Graph Neural Network-driven Transformers [3.805262583092311]
  Neuromorphic vision is a bio-inspired technology that has triggered a paradigm shift in the computer-vision community. Neuromorphic cameras suffer from significant amounts of measurement noise. This noise deteriorates the performance of neuromorphic event-based perception and navigation algorithms.
  arXiv  Detail & Related papers  (2021-12-17T18:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.