STEER: Steerable Dyadic Head Avatars

• URL: http://arxiv.org/abs/2607.23840v1
• Date: Sun, 26 Jul 2026 20:53:41 GMT
• Title: STEER: Steerable Dyadic Head Avatars
• Abstract summary: We present STEER, a controllable 3D dyadic motion prior for reactive conversational head avatars.<n>STEER factorizes conversational behavior into explicit controls for gaze, head rhythm, and emotion, allowing users to steer how an avatar listens, reacts, and engages with a conversation partner.
• Score: 69.46647113012527
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Facial movement and expression are central to face-to-face communication, conveying turn-taking, attention, agreement, and engagement alongside speech. While speech-driven facial animation has made strong progress in lip synchronization and audio-conditioned motion generation, most methods treat conversational behavior as an emergent byproduct of audio, or expose only coarse sequence-level affect control. As a result, key non-verbal channels such as gaze contact and aversion, rhythmic head motion, and emotion remain difficult to explicitly control. We present STEER, a controllable 3D dyadic motion prior for reactive conversational head avatars. STEER factorizes conversational behavior into explicit controls for gaze, head rhythm, and emotion, allowing users to steer how an avatar listens, reacts, and engages with a conversation partner. Since temporally aligned annotations for these behaviors are not available in public dyadic corpora, we introduce a tracking and annotation pipeline that recovers behavioral pseudo-labels from in-the-wild dyadic video. A causal flow-matching transformer then learns partner-aware target motion conditioned on audio, partner motion, emotion and the proposed behavioral controls. We further embed STEER in a photorealistic avatar pipeline by extending a Universal Gaussian Head-Avatar Prior with a learned mapping from tracked parametric motion into its avatar-driving space. This enables controllable animation of high-fidelity Gaussian head avatars without re-training the underlying avatar model. STEER outperforms recent dyadic motion baselines on motion quality, dynamics, and diversity, remains competitive on partner coupling, and enables gaze, head-rhythm, and emotion edits together with an interactive live deployment. We make our code and dataset annotations available at our webpage.

Related papers

• SARAH: Spatially Aware Real-time Agentic Humans [58.32612596034656]
  We present the first real-time, fully causal method for spatially-aware conversational motion, deployable on a streaming VR headset.<n>Given a user's position and dyadic audio, our approach produces full-body motion that aligns gestures with speech while orienting the agent according to the user.<n>We validate our approach on a live VR system, bringing spatially-aware conversational agents to real-time deployment.
  arXiv  Detail & Related papers  (2026-02-20T18:59:35Z)
• JoyAvatar: Unlocking Highly Expressive Avatars via Harmonized Text-Audio Conditioning [18.72712280434528]
  JoyAvatar is a framework capable of generating long duration avatar videos.<n>We introduce a twin-teacher enhanced training algorithm that enables the model to transfer inherent text-controllability.<n>During training, we dynamically modulate the strength of multi-modal conditions.
  arXiv  Detail & Related papers  (2026-01-31T13:00:57Z)
• Avatar Forcing: Real-Time Interactive Head Avatar Generation for Natural Conversation [71.38488610271247]
  Talking head generation creates lifelike avatars from static portraits for virtual communication and content creation.<n>Current models do not yet convey the feeling of truly interactive communication, often generating one-way responses that lack emotional engagement.<n>We propose Avatar Forcing, a new framework for interactive head avatar generation that models real-time user-avatar interactions through diffusion forcing.
  arXiv  Detail & Related papers  (2026-01-02T11:58:48Z)
• AvatarSync: Rethinking Talking-Head Animation through Phoneme-Guided Autoregressive Perspective [15.69417162113696]
  AvatarSync is an autoregressive framework on phoneme representations that generates realistic talking-head animations from a single reference image.<n>We show that AvatarSync outperforms existing talking-head animation methods in visual fidelity, temporal consistency, and computational efficiency.
  arXiv  Detail & Related papers  (2025-09-15T15:34:02Z)
• HM-Talker: Hybrid Motion Modeling for High-Fidelity Talking Head Synthesis [90.74616208952791]
  HM-Talker is a novel framework for generating high-fidelity, temporally coherent talking heads.<n>Explicit cues use Action Units (AUs), anatomically defined facial muscle movements, alongside implicit features to minimize phoneme-viseme misalignment.
  arXiv  Detail & Related papers  (2025-08-14T12:01:52Z)
• MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding [48.54455964043634]
  MEDTalk is a novel framework for fine-grained and dynamic emotional talking head generation.<n>We integrate audio and speech text, predicting frame-wise intensity variations and dynamically adjusting static emotion features to generate realistic emotional expressions.<n>Our generated results can be conveniently integrated into the industrial production pipeline.
  arXiv  Detail & Related papers  (2025-07-08T15:14:27Z)
• Playmate: Flexible Control of Portrait Animation via 3D-Implicit Space Guided Diffusion [6.677873152109559]
  We propose a novel two-stage training framework Playmate to generate more lifelike facial expressions and talking faces.<n>In the first stage, we introduce a decoupled implicit 3D representation along with a meticulously designed motion-decoupled module to facilitate more accurate attribute disentanglement.<n>In the second stage, we introduce an emotion-control module to encode emotion control information into the latent space, enabling fine-grained control over emotions and thereby achieving the ability to generate talking videos with desired emotion.
  arXiv  Detail & Related papers  (2025-02-11T02:53:48Z)
• INFP: Audio-Driven Interactive Head Generation in Dyadic Conversations [11.101103116878438]
  We propose INFP, a novel audio-driven head generation framework for dyadic interaction.<n>INFP comprises a Motion-Based Head Imitation stage and an Audio-Guided Motion Generation stage.<n>To facilitate this line of research, we introduce DyConv, a large scale dataset of rich dyadic conversations collected from the Internet.
  arXiv  Detail & Related papers  (2024-12-05T10:20:34Z)
• InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation [39.235962838952624]
  In this paper, we propose a novel text-guided approach for generating emotionally expressive 2D avatars. Our framework, named InstructAvatar, leverages a natural language interface to control the emotion as well as the facial motion of avatars. Experimental results demonstrate that InstructAvatar produces results that align well with both conditions.
  arXiv  Detail & Related papers  (2024-05-24T17:53:54Z)
• Audio- and Gaze-driven Facial Animation of Codec Avatars [149.0094713268313]
  We describe the first approach to animate Codec Avatars in real-time using audio and/or eye tracking. Our goal is to display expressive conversations between individuals that exhibit important social signals.
  arXiv  Detail & Related papers  (2020-08-11T22:28:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.