FreeTalk: Emotional Topology-Free 3D Talking Heads

• URL: http://arxiv.org/abs/2603.15512v1
• Date: Mon, 16 Mar 2026 16:38:17 GMT
• Title: FreeTalk: Emotional Topology-Free 3D Talking Heads
• Abstract summary: FreeTalk is a two-stage framework for emotion-conditioned 3D talking-head animation.<n>ATS predicts a temporally coherent sequence of 3D landmark displacements from speech audio.<n>Sparse-To-Mesh transfers the predicted landmark motion to a target mesh by combining intrinsic surface features with landmark-to-vertex conditioning.
• Score: 17.706608409435695
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speech-driven 3D facial animation has advanced rapidly, yet most approaches remain tied to registered template meshes, preventing effective deployment on raw 3D scans with arbitrary topology. At the same time, modeling controllable emotional dynamics beyond lip articulation remains challenging, and is often tied to template-based parameterizations. We address these challenges by proposing FreeTalk, a two-stage framework for emotion-conditioned 3D talking-head animation that generalizes to unregistered face meshes with arbitrary vertex count and connectivity. First, Audio-To-Sparse (ATS) predicts a temporally coherent sequence of 3D landmark displacements from speech audio, conditioned on an emotion category and intensity. This sparse representation captures both articulatory and affective motion while remaining independent of mesh topology. Second, Sparse-To-Mesh (STM) transfers the predicted landmark motion to a target mesh by combining intrinsic surface features with landmark-to-vertex conditioning, producing dense per-vertex deformations without template fitting or correspondence supervision at test time. Extensive experiments show that FreeTalk matches specialized baselines when trained in-domain, while providing substantially improved robustness to unseen identities and mesh topologies. Code and pre-trained models will be made publicly available.

Related papers

• RegHead: Non-Humanoid Head Blendshapes via Feed-Forward Registration [64.34831170405567]
  We present RegHead, a framework for constructing semantic blendshape sets for animatable non-humanoid head avatars.<n>Our approach produces higher-fidelity expression meshes than baselines, while running orders of magnitude faster than optimization.
  arXiv  Detail & Related papers  (2026-07-13T23:16:29Z)
• PhyScene3D: Physically Consistent Interactive 3D Tabletop Scene Generation [75.9028457575434]
  Generating 3D tabletop scenes is a fundamental yet underexplored problem for interactive and generalist robotic learning.<n>Here, we introduce PhyScene3D, a framework that reformulates generation as a Human-Mimetic Constructive Process.<n>Experiments demonstrate that PhyScene3D outperforms state-of-the-art approaches in both semantic accuracy and physical validity.
  arXiv  Detail & Related papers  (2026-06-01T03:56:32Z)
• Towards Seamless Interaction: Causal Turn-Level Modeling of Interactive 3D Conversational Head Dynamics [40.86039227407712]
  We present TIMAR (Turn-level Interleaved Masked AutoRegression), a causal framework for 3D conversational head generation.<n>It fuses multimodal information within each turn and applies turn-level causal attention to accumulate conversational history.<n>Experiments on the DualTalk benchmark show that TIMAR reduces Fréchet Distance and MSE by 15-30% on the test set.
  arXiv  Detail & Related papers  (2025-12-17T11:37:35Z)
• Learning Disentangled Speech- and Expression-Driven Blendshapes for 3D Talking Face Animation [20.91704034858042]
  We model facial animation driven by both speech and emotion as a linear additive problem.<n>We learn a set of blendshapes driven by speech and emotion that can be mapped to the expression and jaw pose parameters of the FLAME model.<n>Our approach achieves superior emotional expressivity compared to existing methods, without compromising lip-sync quality.
  arXiv  Detail & Related papers  (2025-10-29T07:29:21Z)
• SeqAffordSplat: Scene-level Sequential Affordance Reasoning on 3D Gaussian Splatting [85.87902260102652]
  We introduce the novel task of Sequential 3D Gaussian Affordance Reasoning.<n>We then propose SeqSplatNet, an end-to-end framework that directly maps an instruction to a sequence of 3D affordance masks.<n>Our method sets a new state-of-the-art on our challenging benchmark, effectively advancing affordance reasoning from single-step interactions to complex, sequential tasks at the scene level.
  arXiv  Detail & Related papers  (2025-07-31T17:56:55Z)
• OT-Talk: Animating 3D Talking Head with Optimal Transportation [20.023346831300373]
  OT-Talk is the first approach to leverage optimal transportation to optimize the learning model in talking head animation.<n>Building on existing learning frameworks, we utilize a pre-trained Hubert model to extract audio features and a transformer model to process temporal sequences.<n>Our experiments on two public audio-mesh datasets demonstrate that our method outperforms state-of-the-art techniques.
  arXiv  Detail & Related papers  (2025-05-03T21:49:23Z)
• GaussianSpeech: Audio-Driven Gaussian Avatars [76.10163891172192]
  We introduce GaussianSpeech, a novel approach that synthesizes high-fidelity animation sequences of photo-realistic, personalized 3D human head avatars from spoken audio.<n>We propose a compact and efficient 3DGS-based avatar representation that generates expression-dependent color and leverages wrinkle- and perceptually-based losses to synthesize facial details.
  arXiv  Detail & Related papers  (2024-11-27T18:54:08Z)
• Beyond Fixed Topologies: Unregistered Training and Comprehensive Evaluation Metrics for 3D Talking Heads [13.003073077799835]
  This work presents a framework capable of animating 3D faces in arbitrary topologies, including real scanned data.<n>Our approach relies on a model leveraging heat diffusion to predict features robust to the mesh topology.
  arXiv  Detail & Related papers  (2024-10-14T19:42:09Z)
• EmoTalk3D: High-Fidelity Free-View Synthesis of Emotional 3D Talking Head [30.138347111341748]
  We present a novel approach for synthesizing 3D talking heads with controllable emotion. Our model enables controllable emotion in the generated talking heads and can be rendered in wide-range views. Experiments demonstrate the effectiveness of our approach in generating high-fidelity and emotion-controllable 3D talking heads.
  arXiv  Detail & Related papers  (2024-08-01T05:46:57Z)
• FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models [85.16273912625022]
  We introduce FaceTalk, a novel generative approach designed for synthesizing high-fidelity 3D motion sequences of talking human heads from audio signal. To the best of our knowledge, this is the first work to propose a generative approach for realistic and high-quality motion synthesis of human heads.
  arXiv  Detail & Related papers  (2023-12-13T19:01:07Z)
• GSmoothFace: Generalized Smooth Talking Face Generation via Fine Grained 3D Face Guidance [83.43852715997596]
  GSmoothFace is a novel two-stage generalized talking face generation model guided by a fine-grained 3d face model. It can synthesize smooth lip dynamics while preserving the speaker's identity. Both quantitative and qualitative experiments confirm the superiority of our method in terms of realism, lip synchronization, and visual quality.
  arXiv  Detail & Related papers  (2023-12-12T16:00:55Z)
• Pose-Controllable 3D Facial Animation Synthesis using Hierarchical Audio-Vertex Attention [52.63080543011595]
  A novel pose-controllable 3D facial animation synthesis method is proposed by utilizing hierarchical audio-vertex attention. The proposed method can produce more realistic facial expressions and head posture movements.
  arXiv  Detail & Related papers  (2023-02-24T09:36:31Z)
• LiP-Flow: Learning Inference-time Priors for Codec Avatars via Normalizing Flows in Latent Space [90.74976459491303]
  We introduce a prior model that is conditioned on the runtime inputs and tie this prior space to the 3D face model via a normalizing flow in the latent space. A normalizing flow bridges the two representation spaces and transforms latent samples from one domain to another, allowing us to define a latent likelihood objective. We show that our approach leads to an expressive and effective prior, capturing facial dynamics and subtle expressions better.
  arXiv  Detail & Related papers  (2022-03-15T13:22:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.