ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reaction

• URL: http://arxiv.org/abs/2609.05506v1
• Date: Sat, 29 Aug 2026 01:48:51 GMT
• Title: ECHO: Dyadic 3D Facial Motion Generation with Asymmetric Deterministic Articulation and Stochastic Reaction
• Abstract summary: We propose ECHO for dyadic 3D facial motion generation under a strict dual-stream audio-only setting.<n>To address this asymmetry, ECHO decomposes motion into a deterministic anchor that captures stable speech-correlated structure.<n>ECHO balances speaking-side articulatory fidelity with listening-side realism and diversity in a single generation process.
• Score: 12.994613743334218
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose ECHO for dyadic 3D facial motion generation under a strict dual-stream audio-only setting, formulating the problem as an asymmetric task involving speech-constrained articulation and one-to-many listener reactions. To address this asymmetry, ECHO decomposes motion into a deterministic anchor that captures stable speech-correlated structure and a stochastic residual that models the remaining one-to-many interaction dynamics. On top of this backbone, Motion Memory acts as a training-only regularizer during brief late-stage fine-tuning to provide local priors for weakly conditioned listening windows, while semantic-group scaling controls residual injection across expression, jaw, and neck. This design balances speaking-side articulatory fidelity with listening-side realism and diversity in a single generation process. Results from unified, state-wise, and ablation evaluations show that conversational 3D motion benefits from decomposing stable and uncertain components rather than applying stochasticity uniformly. ECHO provides a practical formulation and technical basis for deployable conversational digital humans under strict audio-only conditions.

Related papers

• Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding [63.599342750566485]
  SPAR is a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation.<n>Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively.
  arXiv  Detail & Related papers  (2026-08-29T09:58:24Z)
• SubtleTalk: Generating Controllable Weakly-correlated Facial Dynamics for 3D Talking Heads via Residual Flow Matching [21.891013914273714]
  Audio-driven 3D facial animation aims to synthesize realistic and temporally coherent motions from speech.<n>We present SubtleTalk, a framework for generating natural and controllable weakly correlated facial dynamics.<n>Our method significantly improves the realism and diversity of weakly correlated facial dynamics while preserving accurate lip synchronization.
  arXiv  Detail & Related papers  (2026-08-03T11:28:19Z)
• NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction [30.839290792276852]
  Reconstructing continuous speech from scalp electroencephalography (EEG) remains fundamentally challenging.<n>We introduce NeuroSonic, a conditional flow-matching framework for EEG-to-speech reconstruction.<n>NeuroSonic learns a deterministic probability-flow field that transports a noise-corrupted acoustic state toward clean speech under EEG conditioning.
  arXiv  Detail & Related papers  (2026-06-23T03:04:53Z)
• Synthesizing the Lombard Effect: Multi-Level Control of Speech Clarity and Vocal Effort in TTS [61.29502937013759]
  We introduce a flow-matching based text-to-speech model trained with vocal effort and articulation pseudo-labels.<n>The proposed model achieves continuous and disentangled control of vocal effort and articulation, while also enabling word-level emphasis for clarifying specific segments of an utterance.
  arXiv  Detail & Related papers  (2026-06-22T11:14:22Z)
• EmoZone-Talker: Regional Semantic Control of Audio-Driven 3DGS Talking Heads via Facial Action Units [28.454399017725006]
  EmoZone-Talker is a novel framework that reformulates audio-driven facial animation as a structured spatial-temporal coordination problem.<n>Our method improves expression controllability and realism, with notable gains in upper-face accuracy and temporal coherence.
  arXiv  Detail & Related papers  (2026-06-14T15:10:05Z)
• SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models [86.19617358080016]
  Social Omni is a benchmark that operationalizes the evaluation of conversational interactivity across three core dimensions.<n>Social Omni features 2,000 perception samples and a quality-controlled diagnostic set of 209 interaction-generation instances.<n>Our analysis reveals a pronounced decoupling between a model's perceptual accuracy and its ability to generate contextually appropriate interruptions.
  arXiv  Detail & Related papers  (2026-03-17T17:58:44Z)
• FreeTalk: Emotional Topology-Free 3D Talking Heads [17.706608409435695]
  FreeTalk is a two-stage framework for emotion-conditioned 3D talking-head animation.<n>ATS predicts a temporally coherent sequence of 3D landmark displacements from speech audio.<n>Sparse-To-Mesh transfers the predicted landmark motion to a target mesh by combining intrinsic surface features with landmark-to-vertex conditioning.
  arXiv  Detail & Related papers  (2026-03-16T16:38:17Z)
• HM-Talker: Hybrid Motion Modeling for High-Fidelity Talking Head Synthesis [90.74616208952791]
  HM-Talker is a novel framework for generating high-fidelity, temporally coherent talking heads.<n>Explicit cues use Action Units (AUs), anatomically defined facial muscle movements, alongside implicit features to minimize phoneme-viseme misalignment.
  arXiv  Detail & Related papers  (2025-08-14T12:01:52Z)
• High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models [56.00939852727501]
  Minimally-supervised speech synthesis decouples TTS by combining two types of discrete speech representations. Non-autoregressive framework enhances controllability, and duration diffusion model enables diversified prosodic expression.
  arXiv  Detail & Related papers  (2023-09-27T09:27:03Z)
• Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem [65.25725367771075]
  This study demonstrates, for the first time, that the synthesis-based approach can also perform well on this problem. Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols. By utilizing the synthesis model with the input of discrete symbols, after the prediction of discrete symbol sequence, each target speech could be re-synthesized.
  arXiv  Detail & Related papers  (2021-12-17T08:35:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.