Proxy Avatar Meets Low-Rank Caching: Real-Time One-Shot Emotion-Controllable Portrait Animation

• URL: http://arxiv.org/abs/2608.01978v1
• Date: Mon, 03 Aug 2026 09:41:59 GMT
• Title: Proxy Avatar Meets Low-Rank Caching: Real-Time One-Shot Emotion-Controllable Portrait Animation
• Abstract summary: We propose Proxy Avatar Low-Rankching, a framework for real-time one-shot-controllable portrait animation.<n>Our method achieves stronger emotional expressiveness, better identity-preserving animation, and substantially reduced inference cost.
• Score: 33.36600125036859
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio-driven portrait animation has advanced rapidly with diffusion-based generative models, yet real-time one-shot generation with expressive emotion control remains challenging. Existing methods often suffer from insufficient emotion-aware motion priors and expensive appearance computation during multi-step denoising. To address these issues, we propose Proxy Avatar Meets Low-Rank Caching, a cascaded framework for real-time one-shot emotion-controllable portrait animation. Instead of directly generating the target portrait from audio, our method uses a Gaussian-based emotion proxy avatar as a reusable motion generator, which is trained once on a single identity to produce expressive driving videos from audio and emotion labels. Since the proxy avatar only provides motion rather than target appearance or geometry, a large-scale one-shot retargeting model further extracts identity-independent motion from the proxy performance and adapts it to arbitrary target portraits. To improve inference efficiency, we introduce zero-shot appearance reuse with low-rank caching, which caches reference appearance features at the initial denoising step and models subsequent feature variations using lightweight low-rank adapters. Extensive experiments demonstrate that our method achieves stronger emotional expressiveness, better identity-preserving animation, and substantially reduced inference cost, enabling real-time one-shot portrait animation.

Related papers

• IM-Animation: An Implicit Motion Representation for Identity-decoupled Character Animation [58.297199313494]
  Implicit methods capture motion semantics directly from driving video, but suffer from identity leakage and entanglement between motion and appearance.<n>We propose a novel implicit motion representation that compresses per-frame motion into compact 1D motion tokens.<n>Our methodology employs a three-stage training strategy to enhance the training efficiency and ensure high fidelity.
  arXiv  Detail & Related papers  (2026-02-07T11:17:20Z)
• DeX-Portrait: Disentangled and Expressive Portrait Animation via Explicit and Latent Motion Representations [31.845995837468536]
  We propose DeX-Portrait, a novel approach capable of generating expressive portrait animation driven by disentangled pose and expression signals.<n>First, we design a powerful motion trainer to learn both pose and expression encoders for extracting precise and decomposed driving signals.<n>Experiments show that our method outperforms state-of-the-art baselines on both animation quality and disentangled controllability.
  arXiv  Detail & Related papers  (2025-12-17T15:23:57Z)
• PersonaLive! Expressive Portrait Image Animation for Live Streaming [53.63615310186964]
  PersonaLive is a novel diffusion-based framework towards streaming real-time portrait animation.<n>We first adopt hybrid implicit signals, namely implicit facial representations and 3D implicit keypoints, to achieve expressive image-level motion control.<n>Experiments demonstrate that PersonaLive achieves state-of-the-art performance with up to 7-22x speedup over prior diffusion-based portrait animation models.
  arXiv  Detail & Related papers  (2025-12-12T03:24:40Z)
• Audio Driven Real-Time Facial Animation for Social Telepresence [65.66220599734338]
  We present an audio-driven real-time system for animating photorealistic 3D facial avatars with minimal latency.<n>Central to our approach is an encoder model that transforms audio signals into latent facial expression sequences in real time.<n>We capture the rich spectrum of facial expressions necessary for natural communication while achieving real-time performance.
  arXiv  Detail & Related papers  (2025-10-01T17:57:05Z)
• Stable Video-Driven Portraits [52.008400639227034]
  Animation aims to generate photo-realistic videos from a single source image by reenacting the expression and pose from a driving video.<n>Recent advances using diffusion models have demonstrated improved quality but remain constrained by weak control signals and architectural limitations.<n>We propose a novel diffusion based framework that leverages masked facial regions specifically the eyes, nose, and mouth from the driving video as strong motion control cues.
  arXiv  Detail & Related papers  (2025-09-22T08:11:08Z)
• Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animation [72.20148916920944]
  Follow-Your-Emoji-Faster is an efficient diffusion-based framework for portrait animation driven by facial landmarks.<n>Our model supports controllable and expressive animation across diverse portrait types, including real faces, cartoons, sculptures, and animals.<n>EmojiBench++ is a more comprehensive benchmark comprising diverse portraits, driving videos, and landmark sequences.
  arXiv  Detail & Related papers  (2025-09-20T11:09:01Z)
• X-Actor: Emotional and Expressive Long-Range Portrait Acting from Audio [27.619816538121327]
  X-Actor generates lifelike, emotionally expressive talking head videos from a single reference image and an input audio clip.<n>By operating in a compact facial motion latent space decoupled from visual and identity cues, our autoregressive diffusion model effectively captures long-range correlations between audio and facial dynamics.<n>X-Actor produces compelling, cinematic-style performances that go beyond standard talking head animations.
  arXiv  Detail & Related papers  (2025-08-04T22:57:01Z)
• X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention [18.211762995744337]
  We propose X-Portrait, an innovative conditional diffusion model tailored for generating expressive and temporally coherent portrait animation. Given a single portrait as appearance reference, we aim to animate it with motion derived from a driving video, capturing both highly dynamic and subtle facial expressions. Experimental results demonstrate the universal effectiveness of X-Portrait across a diverse range of facial portraits and expressive driving sequences.
  arXiv  Detail & Related papers  (2024-03-23T20:30:28Z)
• GMTalker: Gaussian Mixture-based Audio-Driven Emotional Talking Video Portraits [60.05683966405544]
  We present GMTalker, a Gaussian mixture-based emotional talking portraits generation framework.<n>Specifically, we propose a continuous and disentangled latent space, achieving more flexible emotion manipulation.<n>We also introduce a normalizing flow-based motion generator pretrained on a large dataset to generate diverse head poses, blinks, and eyeball movements.
  arXiv  Detail & Related papers  (2023-12-12T19:03:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.