Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation

• URL: http://arxiv.org/abs/2606.23625v1
• Date: Mon, 22 Jun 2026 17:19:57 GMT
• Title: Learning to See While Learning to Act: Diffusion Models for Active Perception in Robot Imitation
• Abstract summary: We propose See2Act, an imitation learning approach that conditions action prediction on a sequence of actively-inferred viewpoints at test time.<n>The policy is trained using camera poses anchored to actions from offline demonstrations, enabling implicit learning of where to see, while learning how to act.
• Score: 13.13689771374717
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most imitation learning methods assume full observability in table-top settings. In practice, objects are often occluded, requiring robots to both search and act, and learning this coupled behavior from limited demonstrations remains challenging. We propose See2Act, an imitation learning approach that conditions action prediction on a sequence of actively-inferred viewpoints at test time, by coupling action denoising with viewpoint refinement. The policy is trained using camera poses anchored to keyframe actions from offline demonstrations, enabling implicit learning of where to see, while learning how to act. We empirically demonstrate that in Ravens the policy recovers informative viewpoints under severe occlusions, and on RLBench tasks it improves performance by up to 34% over prior methods. In the real world, we collect 50 demonstrations in a digital twin and achieve zero-shot sim-to-real transfer on pick-and-place tasks using depth observations. The policy handles significant occlusions, showing that learned viewpoint reasoning enables robust manipulation under partial observability.

Related papers

• Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations [22.561305437484975]
  We propose Behavior Cloning from Videos via Latent Representations (BCV-LR) to achieve imitation learning from videos.<n>BCV-LR extracts action-related latent features from high-dimensional video inputs through self-supervised tasks.<n>We conduct extensive experiments on a set of challenging visual tasks, including both discrete control and continuous control.
  arXiv  Detail & Related papers  (2025-12-25T09:11:14Z)
• Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration [58.4036440289082]
  Hand-object motion-capture (MoCap) offer large-scale, contact-rich demonstrations and hold promise for dexterous robotic scopes.<n>We introduce Dexplore, a unified single-loop optimization that performs repositories and tracking to learn robot control policies directly from MoCap at scale.
  arXiv  Detail & Related papers  (2025-09-11T17:59:07Z)
• CBIL: Collective Behavior Imitation Learning for Fish from Real Videos [58.81930297206828]
  We present a scalable approach, Collective Behavior Imitation Learning (CBIL), for learning fish schooling behavior directly from videos.<n>MVAE effectively maps 2D observations to implicit states that are compact and expressive for following the imitation learning stage.<n>CBIL can be used for various animation tasks with the learned collective motion priors.
  arXiv  Detail & Related papers  (2025-03-31T21:15:00Z)
• BEAC: Imitating Complex Exploration and Task-oriented Behaviors for Invisible Object Nonprehensile Manipulation [9.453992660558256]
  This paper proposes a novel imitation learning framework called Belief Exploration-Action Cloning (BEAC)<n>Our proposed method achieved the best task performance, higher mode and action prediction accuracies, while reducing the cognitive load in a user study.
  arXiv  Detail & Related papers  (2025-03-21T02:26:14Z)
• Extraneousness-Aware Imitation Learning [25.60384350984274]
  Extraneousness-Aware Learning (EIL) learns visuomotor policies from third-person demonstrations with extraneous subsequences. EIL learns action-conditioned observation embeddings in a self-supervised manner and retrieves task-relevant observations across visual demonstrations. Experimental results show that EIL outperforms strong baselines and achieves comparable policies to those trained with perfect demonstration.
  arXiv  Detail & Related papers  (2022-10-04T04:42:26Z)
• Self-Supervised Learning of Multi-Object Keypoints for Robotic Manipulation [8.939008609565368]
  In this paper, we demonstrate the efficacy of learning image keypoints via the Dense Correspondence pretext task for downstream policy learning. We evaluate our approach on diverse robot manipulation tasks, compare it to other visual representation learning approaches, and demonstrate its flexibility and effectiveness for sample-efficient policy learning.
  arXiv  Detail & Related papers  (2022-05-17T13:15:07Z)
• Action-Conditioned Contrastive Policy Pretraining [39.13710045468429]
  Deep visuomotor policy learning achieves promising results in control tasks such as robotic manipulation and autonomous driving. It requires a huge number of online interactions with the training environment, which limits its real-world application. In this work, we aim to pretrain policy representations for driving tasks using hours-long uncurated YouTube videos.
  arXiv  Detail & Related papers  (2022-04-05T17:58:22Z)
• Playful Interactions for Representation Learning [82.59215739257104]
  We propose to use playful interactions in a self-supervised manner to learn visual representations for downstream tasks. We collect 2 hours of playful data in 19 diverse environments and use self-predictive learning to extract visual representations. Our representations generalize better than standard behavior cloning and can achieve similar performance with only half the number of required demonstrations.
  arXiv  Detail & Related papers  (2021-07-19T17:54:48Z)
• Visual Adversarial Imitation Learning using Variational Models [60.69745540036375]
  Reward function specification remains a major impediment for learning behaviors through deep reinforcement learning. Visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We develop a variational model-based adversarial imitation learning algorithm.
  arXiv  Detail & Related papers  (2021-07-16T00:15:18Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)
• State-Only Imitation Learning for Dexterous Manipulation [63.03621861920732]
  In this paper, we explore state-only imitation learning. We train an inverse dynamics model and use it to predict actions for state-only demonstrations. Our method performs on par with state-action approaches and considerably outperforms RL alone.
  arXiv  Detail & Related papers  (2020-04-07T17:57:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.