CIVIL: Causal and Intuitive Visual Imitation Learning

• URL: http://arxiv.org/abs/2504.17959v3
• Date: Mon, 27 Oct 2025 03:10:19 GMT
• Title: CIVIL: Causal and Intuitive Visual Imitation Learning
• Authors: Yinlong Dai, Robert Ramirez Sanchez, Ryan Jeronimus, Shahabedin Sagheb, Cara M. Nunez, Heramb Nemlekar, Dylan P. Losey,
• Abstract summary: We propose a new training paradigm for robots.<n>Instead of asking humans to show what actions the robot should take, we also enable humans to intuitively indicate why they made those decisions.<n>Our simulations and real-world experiments demonstrate that robots trained with CIVIL learn both what actions to take and why to take those actions.
• Score: 5.090234440090935
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Today's robots attempt to learn new tasks by imitating human examples. These robots watch the human complete the task, and then try to match the actions taken by the human expert. However, this standard approach to visual imitation learning is fundamentally limited: the robot observes what the human does, but not why the human chooses those behaviors. Without understanding which features of the system or environment factor into the human's decisions, robot learners often misinterpret the human's examples. In practice, this results in causal confusion, inefficient learning, and robot policies that fail when the environment changes. We therefore propose a shift in perspective: instead of asking human teachers just to show what actions the robot should take, we also enable humans to intuitively indicate why they made those decisions. Under our paradigm human teachers attach markers to task-relevant objects and use natural language prompts to describe their state representation. Our proposed algorithm, CIVIL, leverages this augmented demonstration data to filter the robot's visual observations and extract a feature representation that aligns with the human teacher. CIVIL then applies these causal features to train a transformer-based policy that -- when tested on the robot -- is able to emulate human behaviors without being confused by visual distractors or irrelevant items. Our simulations and real-world experiments demonstrate that robots trained with CIVIL learn both what actions to take and why to take those actions, resulting in better performance than state-of-the-art baselines. From the human's perspective, our user study reveals that this new training paradigm actually reduces the total time required for the robot to learn the task, and also improves the robot's performance in previously unseen scenarios. See videos at our project website: https://civil2025.github.io

Related papers

• EgoZero: Robot Learning from Smart Glasses [54.6168258133554]
  EgoZero learns robust manipulation policies from human demonstrations captured with Project Aria smart glasses.<n>We deploy EgoZero policies on a Franka Panda robot and demonstrate zero-shot transfer with 70% success rate over 7 manipulation tasks.<n>Our results suggest that in-the-wild human data can serve as a scalable foundation for real-world robot learning.
  arXiv  Detail & Related papers  (2025-05-26T17:59:17Z)
• UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations [24.232732907295194]
  UniSkill is a framework that learns embodiment-agnostic skill representations from large-scale cross-embodiment video data without any labels.<n>Our experiments in both simulation and real-world environments show that our cross-embodiment skills successfully guide robots in selecting appropriate actions, even with unseen video prompts.
  arXiv  Detail & Related papers  (2025-05-13T17:59:22Z)
• What Matters to You? Towards Visual Representation Alignment for Robot Learning [81.30964736676103]
  When operating in service of people, robots need to optimize rewards aligned with end-user preferences. We propose Representation-Aligned Preference-based Learning (RAPL), a method for solving the visual representation alignment problem.
  arXiv  Detail & Related papers  (2023-10-11T23:04:07Z)
• SACSoN: Scalable Autonomous Control for Social Navigation [62.59274275261392]
  We develop methods for training policies for socially unobtrusive navigation. By minimizing this counterfactual perturbation, we can induce robots to behave in ways that do not alter the natural behavior of humans in the shared space. We collect a large dataset where an indoor mobile robot interacts with human bystanders.
  arXiv  Detail & Related papers  (2023-06-02T19:07:52Z)
• Affordances from Human Videos as a Versatile Representation for Robotics [31.248842798600606]
  We train a visual affordance model that estimates where and how in the scene a human is likely to interact. The structure of these behavioral affordances directly enables the robot to perform many complex tasks. We show the efficacy of our approach, which we call VRB, across 4 real world environments, over 10 different tasks, and 2 robotic platforms operating in the wild.
  arXiv  Detail & Related papers  (2023-04-17T17:59:34Z)
• Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning [54.636562516974884]
  In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. In this work, we propose MEDAL++, a novel design for self-improving robotic systems. The robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
  arXiv  Detail & Related papers  (2023-03-02T18:51:38Z)
• Open-World Object Manipulation using Pre-trained Vision-Language Models [72.87306011500084]
  For robots to follow instructions from people, they must be able to connect the rich semantic information in human vocabulary. We develop a simple approach, which leverages a pre-trained vision-language model to extract object-identifying information. In a variety of experiments on a real mobile manipulator, we find that MOO generalizes zero-shot to a wide range of novel object categories and environments.
  arXiv  Detail & Related papers  (2023-03-02T01:55:10Z)
• Aligning Robot and Human Representations [50.070982136315784]
  We argue that current representation learning approaches in robotics should be studied from the perspective of how well they accomplish the objective of representation alignment. We mathematically define the problem, identify its key desiderata, and situate current methods within this formalism.
  arXiv  Detail & Related papers  (2023-02-03T18:59:55Z)
• Learning Latent Representations to Co-Adapt to Humans [12.71953776723672]
  Non-stationary humans are challenging for robot learners. In this paper we introduce an algorithmic formalism that enables robots to co-adapt alongside dynamic humans.
  arXiv  Detail & Related papers  (2022-12-19T16:19:24Z)
• Learning Reward Functions for Robotic Manipulation by Observing Humans [92.30657414416527]
  We use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies. The learned rewards are based on distances to a goal in an embedding space learned using a time-contrastive objective.
  arXiv  Detail & Related papers  (2022-11-16T16:26:48Z)
• Learning Preferences for Interactive Autonomy [1.90365714903665]
  This thesis is an attempt towards learning reward functions from human users by using other, more reliable data modalities. We first propose various forms of comparative feedback, e.g., pairwise comparisons, best-of-many choices, rankings, scaled comparisons; and describe how a robot can use these various forms of human feedback to infer a reward function.
  arXiv  Detail & Related papers  (2022-10-19T21:34:51Z)
• Reasoning about Counterfactuals to Improve Human Inverse Reinforcement Learning [5.072077366588174]
  Humans naturally infer other agents' beliefs and desires by reasoning about their observable behavior. We propose to incorporate the learner's current understanding of the robot's decision making into our model of human IRL. We also propose a novel measure for estimating the difficulty for a human to predict instances of a robot's behavior in unseen environments.
  arXiv  Detail & Related papers  (2022-03-03T17:06:37Z)
• Feature Expansive Reward Learning: Rethinking Human Input [31.413656752926208]
  We introduce a new type of human input in which the person guides the robot from states where the feature being taught is highly expressed to states where it is not. We propose an algorithm for learning the feature from the raw state space and integrating it into the reward function.
  arXiv  Detail & Related papers  (2020-06-23T17:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.