Simulating Infant First-Person Sensorimotor Experience via Motion Retargeting from Babies to Humanoids

• URL: http://arxiv.org/abs/2604.27583v1
• Date: Thu, 30 Apr 2026 08:37:46 GMT
• Title: Simulating Infant First-Person Sensorimotor Experience via Motion Retargeting from Babies to Humanoids
• Abstract summary: We present a framework for simulating the sensorimotor experiences of infants using physical and virtual humanoids.<n>We map the reconstructed motion onto several developmental platforms: the physical iCub robot and the virtual simulators pyCub, EMFANT and MIMo.<n>This framework provides a unique window into the infant's sensorimotor experience, offering new tools for robotics, developmental science, and early detection of neurodevelopmental disorders.
• Score: 8.972726623469823
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Motion retargeting from humans to human-like artificial agents is becoming increasingly important as humanoid robots grow more capable. However, most existing approaches focus only on reproducing kinematics and ignore the rich sensorimotor experience associated with human movement. In this work, we present a framework for simulating the multimodal sensorimotor experiences of infants using physical and virtual humanoids. From a single video, our method reconstructs the infant's body configuration by extracting its skeletal structure and estimating the full 3D pose from each frame. Then we map the reconstructed motion onto several developmental platforms: the physical iCub robot and the virtual simulators pyCub, EMFANT and MIMo. Replaying the retargeted motions on these embodiments produces simulated multisensory streams including proprioception (joints and muscles), touch, and vision. For the best-matching embodiment, the retargeting achieves sub-centimeter accuracy and enables a rich multimodal analysis of infant development as well as enhanced automated annotation of behaviors. This framework provides a unique window into the infant's sensorimotor experience, offering new tools for robotics, developmental science, and early detection of neurodevelopmental disorders. The code is available at https://github.com/ctu-vras/motion-retargeting/.

Related papers

• THRIVE: Therapeutic Humanoid Robot In Virtual Environment [2.9600025934584235]
  THRIVE is an at-home rehabilitation platform that integrates a suite of virtual-reality upper-body rehabilitation games, a real-time camera-based motion-tracking system, and a socially interactive robot therapist.<n>The system is designed for therapy and intervention in children with upper-limb motor impairments.
  arXiv  Detail & Related papers  (2026-08-14T16:42:58Z)
• Embodiment Shapes Rolling Behavior in a Multimodal Infant Model [3.40533661316776]
  Rolling over is one of the earliest milestones in infant motor development.<n>We conduct a computational study of infant rolling using MIMo, a virtual infant embodiment equipped with proprioception and vestibular sensation.
  arXiv  Detail & Related papers  (2026-06-16T03:18:45Z)
• GRAIL: Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors [113.71148915419246]
  We present GRAIL, a digital generation pipeline that composes 3D assets, simulator-ready scenes, and priors from video foundation models (VFMs) to synthesize interactions without rebuilding physical environments or teleoperating the robot.<n>GRAIL starts from fully specified 3D configurations in which object geometry, camera parameters, metric scale, environment depth, and a robot-proportioned character are known before video generation and reused during reconstruction.<n>We retarget the recovered motions to a humanoid robot and train complementary task-general morphology trackers.<n>GRAIL produces over 20,000 sequences spanning pick-up, object manipulation, sitting
  arXiv  Detail & Related papers  (2026-06-03T17:57:45Z)
• MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction [54.36564144414704]
  MeshMimic is an innovative framework that bridges 3D scene reconstruction and embodied intelligence to enable humanoid robots to learn coupled "motion-terrain" interactions directly from video.<n>By leveraging state-of-the-art 3D vision models, our framework precisely segments and reconstructs both human trajectories and the underlying 3D geometry of terrains and objects.
  arXiv  Detail & Related papers  (2026-02-17T17:09:45Z)
• FRoM-W1: Towards General Humanoid Whole-Body Control with Language Instructions [147.04372611893032]
  We present FRoM-W1, an open-source framework designed to achieve general humanoid whole-body motion control using natural language.<n>We extensively evaluate FRoM-W1 on Unitree H1 and G1 robots.<n>Results demonstrate superior performance on the HumanML3D-X benchmark for human whole-body motion generation.
  arXiv  Detail & Related papers  (2026-01-19T07:59:32Z)
• From Generated Human Videos to Physically Plausible Robot Trajectories [103.28274349461607]
  Video generation models are rapidly improving in their ability to synthesize human actions in novel contexts.<n>To realize this potential, how can a humanoid execute the human actions from generated videos in a zero-shot manner?<n>This challenge arises because generated videos are often noisy and exhibit morphological distortions that make direct imitation difficult compared to real video.<n>We propose GenMimic, a physics-aware reinforcement learning policy conditioned on 3D keypoints, and trained with symmetry regularization and keypoint-weighted tracking rewards.
  arXiv  Detail & Related papers  (2025-12-04T18:56:03Z)
• Implicit Kinodynamic Motion Retargeting for Human-to-humanoid Imitation Learning [35.8296790596745]
  Implicit Kinodynamic Motion Retargeting (IKMR) is a novel efficient and scalable framework that considers both kinematics and dynamics.<n>IKMR pretrains motion topology representation and a dual encoder-decoder architecture to learn a motion domain mapping.<n>We conduct our experiments both in the simulator and the real robot on a full-size humanoid robot.
  arXiv  Detail & Related papers  (2025-09-18T21:34:02Z)
• A computational model of infant sensorimotor exploration in the mobile paradigm [13.666777211441286]
  We present a computational model of the mechanisms that may determine infants' behavior in the "mobile paradigm"<n>In this paradigm, a mobile is connected to one of the infant's limbs, prompting the infant to preferentially move that "connected" limb.<n>Our model incorporates a neural network, action-outcome prediction, exploration, motor noise, preferred activity level, and biologically-inspired motor control.
  arXiv  Detail & Related papers  (2025-04-24T21:02:06Z)
• PRIMAL: Physically Reactive and Interactive Motor Model for Avatar Learning [38.004463823796286]
  We formulate the motor system of an interactive avatar as a generative motion model.<n>Inspired by recent advances in foundation models, we propose PRIMAL.<n>We leverage the model to create a real-time character animation system in Unreal Engine that feels highly responsive and natural.
  arXiv  Detail & Related papers  (2025-03-21T21:27:57Z)
• Universal Humanoid Motion Representations for Physics-Based Control [71.46142106079292]
  We present a universal motion representation that encompasses a comprehensive range of motor skills for physics-based humanoid control. We first learn a motion imitator that can imitate all of human motion from a large, unstructured motion dataset. We then create our motion representation by distilling skills directly from the imitator.
  arXiv  Detail & Related papers  (2023-10-06T20:48:43Z)
• Robot Learning with Sensorimotor Pre-training [98.7755895548928]
  We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.
  arXiv  Detail & Related papers  (2023-06-16T17:58:10Z)
• Learning body models: from humans to humanoids [2.855485723554975]
  Humans and animals excel in combining information from multiple sensory modalities, controlling their complex bodies, adapting to growth, failures, or using tools. Key foundation is an internal representation of the body that the agent - human, animal, or robot - has developed. mechanisms of operation of body models in the brain are largely unknown and even less is known about how they are constructed from experience after birth.
  arXiv  Detail & Related papers  (2022-11-06T07:30:01Z)
• Modeling Social Interaction for Baby in Simulated Environment for Developmental Robotics [1.9576223566777229]
  We present Simulated Environment for Developmental Robotics (SEDRo) SEDRo simulates the environments for a baby agent that a human baby experiences throughout the pre-born fetus stage to post-birth 12 months. To evaluate different developmental milestones of the agent, SEDRo incorporates some experiments from developmental psychology.
  arXiv  Detail & Related papers  (2020-12-29T16:42:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.