InterSing: Explicit Interaction Dynamics for 3D Duet Singing Animation and Beyond

• URL: http://arxiv.org/abs/2609.04903v1
• Date: Fri, 04 Sep 2026 09:02:52 GMT
• Title: InterSing: Explicit Interaction Dynamics for 3D Duet Singing Animation and Beyond
• Abstract summary: InterSing is a framework for generating realistic 3D head animations for duet singing performances.<n>InterSing generates realistic and expressive singing head animations with stronger coordination and musical alignment than existing methods.
• Score: 56.764269093059504
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present InterSing, a framework for generating realistic 3D head animations for duet singing performances. Unlike solo singing, duet performance requires each singer to balance individual expressiveness with intermittent interaction at musically salient moments, such as phrase boundaries, synchronized rhythms, and call-and-response passages. Because these interactions are sparse and rhythm-dependent, existing audio-driven animation methods and conversational interaction models do not adequately capture their structure. Our key insight is that duet coordination can be represented as a time-varying signal that reflects how strongly performers engage with one another throughout a song. Based on this observation, we introduce interaction logits, an interpretable latent representation that models the degree of cross-performer engagement at each time step. We learn these logits using weak supervision and use them to condition an interaction-aware diffusion model jointly driven by audio features and interaction dynamics. This formulation enables unified multi-mode generation, spanning independent motion, coordinated behavior, and smooth transitions between them. Experiments show that InterSing generates realistic and expressive singing head animations with stronger coordination and musical alignment than existing methods, while preserving each performer's characteristic motion style. We further demonstrate that the same formulation generalizes to multi-singer performances and provides intuitive control over when and how performers engage.

Related papers

• CETalk: Continuous Valence-Arousal Control for Audio-Driven 3D Talking Head Generation [18.841894940869157]
  We propose CETalk, an audio-driven 3D facial animation framework conditioned on Valence--Arousal representations for fine-grained emotion control.<n>We show that CETalk outperforms state-of-the-art methods in both lip-sync accuracy and emotional expressiveness.
  arXiv  Detail & Related papers  (2026-08-15T08:22:50Z)
• ReFree: Towards Realistic Co-Speech Video Generation via Reward-Free RL and Multilevel Speech Guidance [50.482989497576476]
  Speech-driven talking character animation seeks to generate life-like portrait videos that convey natural conversation behavior.<n>Existing approaches typically trade off precise phoneme-to-lip synchronization against dynamic facial expressions and head motion.<n>We propose ReFree-S2V, a flow-matching speech-to-portrait animation framework that builds upon a pretrained video generation model.
  arXiv  Detail & Related papers  (2026-06-11T13:00:58Z)
• SalsaAgent: A multimodal embodied language model for interactive dance generation [0.7952582509792973]
  We present SalsaAgent, a language model that generates expressive, full-body salsa dance motions in reaction to a human leader.<n>We formulate interaction as nonverbal motion token passing, extending the vocabulary of a large language model.<n>Our contributions include new tokens for full-body and motion relations, LLM fine-tuning, and a two-stage token-to-diffusion pipeline.
  arXiv  Detail & Related papers  (2026-05-28T01:08:27Z)
• MIBURI: Towards Expressive Interactive Gesture Synthesis [62.45332399212876]
  Embodied Conversational Agents (ECAs) aim to emulate human face-to-face interaction through speech, gestures, and facial expressions.<n>Existing solutions for ECAs produce rigid, low-diversity motions that are unsuitable for human-like interaction.<n>We present MIBURI, the first online, causal framework for generating expressive full-body gestures and facial expressions synchronized with real-time spoken dialogue.
  arXiv  Detail & Related papers  (2026-03-03T18:59:51Z)
• InterAct: A Large-Scale Dataset of Dynamic, Expressive and Interactive Activities between Two People in Daily Scenarios [40.42003202491803]
  We propose to simultaneously model two people's activities, and target objective-driven, dynamic, and semantically consistent interactions.<n>We capture a new multi-modal dataset dubbed InterAct composed of 241 motion sequences.<n>InterAct contains diverse and complex motions of individuals and interesting and relatively long-term interaction patterns barely seen before.
  arXiv  Detail & Related papers  (2025-09-06T15:36:47Z)
• Think2Sing: Orchestrating Structured Motion Subtitles for Singing-Driven 3D Head Animation [69.50178144839275]
  Singing involves richer emotional nuance, dynamic prosody, and lyric-based semantics.<n>Existing speech-driven approaches often produce oversimplified, emotionally flat, and semantically inconsistent results.<n>Think2Sing generates semantically coherent and temporally consistent 3D head animations conditioned on both lyrics and acoustics.
  arXiv  Detail & Related papers  (2025-09-02T12:59:27Z)
• InterDance:Reactive 3D Dance Generation with Realistic Duet Interactions [67.37790144477503]
  We propose InterDance, a large-scale duet dance dataset that significantly enhances motion quality, data scale, and the variety of dance genres.<n>We introduce a diffusion-based framework with an interaction refinement guidance strategy to optimize the realism of interactions progressively.
  arXiv  Detail & Related papers  (2024-12-22T11:53:51Z)
• Dyadic Interaction Modeling for Social Behavior Generation [6.626277726145613]
  We present an effective framework for creating 3D facial motions in dyadic interactions. The heart of our framework is Dyadic Interaction Modeling (DIM), a pre-training approach. Experiments demonstrate the superiority of our framework in generating listener motions.
  arXiv  Detail & Related papers  (2024-03-14T03:21:33Z)
• ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions [66.87211993793807]
  We present ReMoS, a denoising diffusion based model that synthesizes full body motion of a person in two person interaction scenario. We demonstrate ReMoS across challenging two person scenarios such as pair dancing, Ninjutsu, kickboxing, and acrobatics. We also contribute the ReMoCap dataset for two person interactions containing full body and finger motions.
  arXiv  Detail & Related papers  (2023-11-28T18:59:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.