Co-policy: Responsive Human-Robot Co-Creation for Musical Performances

• URL: http://arxiv.org/abs/2606.19914v1
• Date: Thu, 18 Jun 2026 08:08:16 GMT
• Title: Co-policy: Responsive Human-Robot Co-Creation for Musical Performances
• Abstract summary: Co-policy is a framework for human-robot musical co-creation.<n>It separates semantic intent grounding, constrained musical variation, and visuomotor execution.<n>We show improved intent alignment, execution accuracy, and response frequency over diffusion-policy and ablated baselines.
• Score: 58.92814200971293
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Art has long stood as a pivotal expression of human creativity. Embodied artificial intelligence offers a route for generative models to participate in that creativity through physical action rather than disembodied digital content. In robotic music co-creation, it is challenging to connect semantic musical understanding with real-time and physically executable performance. We present Co-policy, a framework for human-robot musical co-creation that separates semantic intent grounding, constrained musical variation, and visuomotor execution. To ground musical semantics, Co-policy uses pre-inference semantic anchors and a fine-tuned Qwen-vl planner (F-Qwen) to transform speech, live musical seeds, and visual observations into structured co-creation plans. To support low-latency execution, Co-policy introduces a Gaussian-Mixture Visuomotor Policy (GMP), implemented as a conditional mixture-density policy that maps target notes and visual context to multimodal robot actions in a single forward pass. Unlike robotic playback systems that merely reproduce user-specified notes, Co-policy generates complementary musical responses under both musical and physical constraints. Real-robot chime experiments, ablations, and expert evaluation show improved intent alignment, execution accuracy, and response frequency over diffusion-policy and ablated baselines, supporting physically grounded action generation as a key requirement for embodied human-AI co-creation.

Related papers

• RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction [33.04319504310729]
  RoboGesture is a robot-centric framework that co-designs data, modeling, and control to power a complete interactive human-humanoid system.<n>We first establish the RoboGesture dataset featuring over 300 gesture categories and develop an automated pipeline to synthesize large-scale audio-motion pairs.<n>Experiments on a Unitree G1 humanoid demonstrate that RoboGesture generates safer, more rhythmic, and more semantically appropriate responses compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-08-27T02:13:46Z)
• Semantic Audio-driven Understanding for Dynamic Humanoid Whole Body Control [0.0]
  We introduce a novel multi-modal orchestration framework for semantic audio-driven humanoid control.<n>The system processes continuous audio streams and routes them into music or speech branches.<n>Music input is handled via audio fingerprinting and semantic embeddings to retrieve track identity and temporal alignment.<n>Speech input is grounded into a discrete library of imitation-learned skills, enabling direct human-robot interaction.
  arXiv  Detail & Related papers  (2026-07-15T14:24:00Z)
• Selective Imperfection as a Generative Framework for Analysis, Creativity and Discovery [1.3537117504260623]
  We show how sound functions as a scientific probe, an inversion where listening becomes a mode of seeing and musical composition becomes a blueprint for matter.<n>We show that science and art are generative acts of world-building under constraint, with vibration as a shared grammar organizing structure across scales.
  arXiv  Detail & Related papers  (2025-12-30T11:14:51Z)
• Do You Have Freestyle? Expressive Humanoid Locomotion via Audio Control [52.83779852397341]
  Current humanoid robots lack expressive improvisational capabilities, confined to predefined motions or sparse commands.<n>We propose RoboPerform, the first unified audio-to-locomotion framework that can directly generate music-driven dance and speech-driven co-speech gestures from audio.<n>RoboPerform achieves a ResMoE policy for adapting to diverse motion patterns and a diffusion-based student policy for audio style injection.
  arXiv  Detail & Related papers  (2025-12-29T17:59:24Z)
• Semantic Co-Speech Gesture Synthesis and Real-Time Control for Humanoid Robots [5.531678625546847]
  We present an innovative end-to-end framework for synthesizing semantically meaningful co-speech gestures and deploying them in real-time on a humanoid robot.<n>Our core innovation lies in the meticulous integration of a semantics-aware gesture synthesis module.<n>We demonstrate that our combined system produces semantically appropriate and rhythmically coherent gestures.
  arXiv  Detail & Related papers  (2025-12-19T02:55:10Z)
• The Ghost in the Keys: A Disklavier Demo for Human-AI Musical Co-Creativity [59.78509280246215]
  Aria-Duet is an interactive system facilitating a real-time musical duet between a human pianist and Aria, a state-of-the-art generative model.<n>We analyze the system's output from a musicological perspective, finding the model can maintain stylistic semantics and develop coherent phrasal ideas.
  arXiv  Detail & Related papers  (2025-11-03T15:26:01Z)
• PhysHSI: Towards a Real-World Generalizable and Natural Humanoid-Scene Interaction System [67.2851799763138]
  PhysHSI comprises a simulation training pipeline and a real-world deployment system.<n>In simulation, we adopt adversarial motion prior-based policy learning to imitate natural humanoid-scene interaction data.<n>For real-world deployment, we introduce a coarse-to-fine object localization module that combines LiDAR and camera inputs.
  arXiv  Detail & Related papers  (2025-10-13T07:11:37Z)
• Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesis [51.95817740348585]
  Human-X is a novel framework designed to enable immersive and physically plausible human interactions across diverse entities.<n>Our method jointly predicts actions and reactions in real-time using an auto-regressive reaction diffusion planner.<n>Our framework is validated in real-world applications, including virtual reality interface for human-robot interaction.
  arXiv  Detail & Related papers  (2025-08-04T06:35:48Z)
• Flat latent manifolds for music improvisation between human and machine [9.571383193449648]
  We consider a music-generating algorithm as a counterpart to a human musician, in a setting where reciprocal improvisation is to lead to new experiences. In the learned model, we generate novel musical sequences by quantification in latent space. We provide empirical evidence for our method via a set of experiments on music and we deploy our model for an interactive jam session with a professional drummer.
  arXiv  Detail & Related papers  (2022-02-23T09:00:17Z)
• RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement Learning [69.20460466735852]
  This paper presents a deep reinforcement learning algorithm for online accompaniment generation. The proposed algorithm is able to respond to the human part and generate a melodic, harmonic and diverse machine part.
  arXiv  Detail & Related papers  (2020-02-08T03:53:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.