Towards Flexible, Natural, Efficient Interaction for Conversational Talking Face Generation

• URL: http://arxiv.org/abs/2606.31088v1
• Date: Tue, 30 Jun 2026 03:27:23 GMT
• Title: Towards Flexible, Natural, Efficient Interaction for Conversational Talking Face Generation
• Abstract summary: We introduce InterTalk, a novel framework designed for highly interactive conversational talking face generation.<n>Our method achieves strong flexibility by explicitly modeling multi-round conversational dynamics among each participant.<n>InterTalk achieves superior interaction quality while maintaining real-time performance at 30 FPS.
• Score: 34.16949333353474
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conversational talking face generation has recently attracted increasing attention, aiming to synthesize interactive talking videos where characters speak, listen, and respond dynamically to each other. This task presents three core challenges: 1) Flexibility: enabling multi-round dialogues with an arbitrary number of participants; 2) Naturalness: maintaining coherent motion and appropriate non-verbal feedback throughout the interaction; and 3) Efficiency: achieving real-time generation and low computation overhead for long-term continuous online conversation. Despite recent advances, existing methods still fall short in balancing all three requirements. To bridge this gap, we introduce InterTalk, a novel and efficient framework designed for highly interactive conversational talking face generation. Built upon a motion-based architecture, InterTalk supports real-time conversation synthesis. Our method achieves strong flexibility by explicitly modeling multi-round conversational dynamics among each participant, eliminating constraints on their numbers. To enhance interactivity, we incorporate motion feedback from multiple participants and introduce an iterative generation strategy for more natural behaviors. Besides, we disentangle motion into several facial components, enabling targeted refinements for natural response such as precise lip sync and realistic eye blinking. Finally, we construct a new multi-person conversational dataset and enrich it with 3D face-based data augmentation. Extensive experiments demonstrate that InterTalk achieves superior interaction quality while maintaining real-time performance at 30 FPS.

Related papers

• Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models [53.470209949659115]
  We propose a post-training alignment method that improves full spoken dialogue models through RL.<n>We address the four canonical axes of interactivity: handling turn-taking, back-channel pause, and user interruption.<n>An extra-based reward for response quality prevents semantic degradation.
  arXiv  Detail & Related papers  (2026-06-09T17:46:55Z)
• MIBURI: Towards Expressive Interactive Gesture Synthesis [62.45332399212876]
  Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions.<n>Existing solutions for ECAs produce rigid, low-diversity motions that are unsuitable for human-like interaction.<n>We present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue.
  arXiv  Detail & Related papers  (2026-03-03T18:59:51Z)
• TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation [72.46711449668814]
  We introduce TAVID, a unified framework that generates both interactive faces and conversational speech in a synchronized manner.<n>We evaluate our system across four dimensions: talking face realism, listening head responsiveness, dyadic interaction, and speech quality.
  arXiv  Detail & Related papers  (2025-12-23T12:04:23Z)
• Interact2Ar: Full-Body Human-Human Interaction Generation via Autoregressive Diffusion Models [80.28579390566298]
  We introduce Interact2Ar, a text-conditioned autoregressive diffusion model for generating full-body, human-human interactions.<n>Hand kinematics are incorporated through dedicated parallel branches, enabling high-fidelity full-body generation.<n>Our model enables a series of downstream applications, including temporal motion composition, real-time adaptation to disturbances, and extension beyond dyadic to multi-person scenarios.
  arXiv  Detail & Related papers  (2025-12-22T18:59:50Z)
• Towards Seamless Interaction: Causal Turn-Level Modeling of Interactive 3D Conversational Head Dynamics [40.86039227407712]
  We present TIMAR (Turn-level Interleaved Masked AutoRegression), a causal framework for 3D conversational head generation.<n>It fuses multimodal information within each turn and applies turn-level causal attention to accumulate conversational history.<n>Experiments on the DualTalk benchmark show that TIMAR reduces Fréchet Distance and MSE by 15-30% on the test set.
  arXiv  Detail & Related papers  (2025-12-17T11:37:35Z)
• Text2Interact: High-Fidelity and Diverse Text-to-Two-Person Interaction Generation [39.67266918328847]
  We propose Text2 framework designed to generate realistic text human-human interactions.<n>We present InterCompose, a synthesis-by-composition pipeline that aligns interaction descriptions with strong singleperson motion priors.<n>We also propose InterActor, a text-to-interaction model with word-level conditioning that preserves token-level cues.
  arXiv  Detail & Related papers  (2025-10-07T22:41:23Z)
• Aligning Spoken Dialogue Models from User Interactions [55.192134724622235]
  We propose a novel preference alignment framework to improve spoken dialogue models on realtime conversations from user interactions.<n>We create a dataset of more than 150,000 preference pairs from raw multi-turn speech conversations annotated with AI feedback.<n>Our findings shed light on the importance of a well-calibrated balance among various dynamics, crucial for natural real-time speech dialogue systems.
  arXiv  Detail & Related papers  (2025-06-26T16:45:20Z)
• DualTalk: Dual-Speaker Interaction for 3D Talking Head Conversations [18.419225973482423]
  Existing 3D talking head generation models focus solely on speaking or listening.<n>We propose a new task -- multi-round dual-speaker interaction for 3D talking head generation.<n>We introduce DualTalk, a novel unified framework that integrates the dynamic behaviors of speakers and listeners.
  arXiv  Detail & Related papers  (2025-05-23T16:49:05Z)
• It Takes Two: Real-time Co-Speech Two-person's Interaction Generation via Reactive Auto-regressive Diffusion Model [34.94330722832987]
  We introduce an audio-driven, auto-regressive system designed to synthesize dynamic movements for two characters during a conversation.<n>To the best of our knowledge, this is the first system capable of generating interactive full-body motions for two characters from speech in an online manner.
  arXiv  Detail & Related papers  (2024-12-03T12:31:44Z)
• A Unified Framework for Motion Reasoning and Generation in Human Interaction [28.736843383405603]
  We introduce Versatile Interactive Motion-language model, which integrates both language and motion modalities.<n>VIM is capable of simultaneously understanding and generating both motion and text modalities.<n>We evaluate VIM across multiple interactive motion-related tasks, including motion-to-text, text-to-motion, reaction generation, motion editing, and reasoning about motion sequences.
  arXiv  Detail & Related papers  (2024-10-08T02:23:53Z)
• Interactive Conversational Head Generation [68.76774230274076]
  We introduce a new conversation head generation benchmark for synthesizing behaviors of a single interlocutor in a face-to-face conversation. The capability to automatically synthesize interlocutors which can participate in long and multi-turn conversations is vital and offer benefits for various applications.
  arXiv  Detail & Related papers  (2023-07-05T08:06:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.