Behavior Prompting Policy: Demonstrations as Prompts for Manipulation

• URL: http://arxiv.org/abs/2606.30457v1
• Date: Mon, 29 Jun 2026 15:23:44 GMT
• Title: Behavior Prompting Policy: Demonstrations as Prompts for Manipulation
• Abstract summary: We study behavior prompting, a paradigm that enables robots to perform new tasks at inference time given a single human demonstration.<n>We present contributions in algorithm, data, and evaluation.
• Score: 12.69446064622751
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study behavior prompting, a paradigm that enables robots to perform new tasks at inference time given a single human demonstration, which we call a behavior prompt. To enable this capability, we present contributions in algorithm, data, and evaluation. For algorithm, we introduce Behavior Prompting Policy (BPP), an in-context visuomotor architecture that translates the behavior prompt and the current observation into robot actions. For data, we identify that task diversity is the primary driver of the prompting capability and introduce iPhUMI, a handheld manipulation interface for collecting diverse training data. For evaluation, we introduce DrawAnything and LIBERO-Gen to evaluate test-time adaptation to unseen drawing and tabletop manipulation tasks. We also demonstrate that iPhUMI serves as a practical interface for specifying behavior prompts at test time, enabling a human to command a robot via a single demonstration to complete known tasks or to define new robot capabilities. Altogether, behavior prompting provides a flexible and scalable way to teach robots new skills without the need for expensive fine-tuning. Our project website is located at https://behavior-prompting.github.io/ .

Related papers

• Imitating What Works: Simulation-Filtered Modular Policy Learning from Human Videos [56.510263910611684]
  We tackle prehensile manipulation, in which tasks involve grasping an object before performing various post-grasp motions.<n>Human videos offer strong signals for learning the post-grasp motions, but they are less useful for learning the prerequisite grasping behaviors.<n>We present Perceive-Simulate-Imitate (PSI), a framework for training a modular manipulation policy using human video motion data.
  arXiv  Detail & Related papers  (2026-02-13T18:59:10Z)
• Feasibility-aware Imitation Learning from Observations through a Hand-mounted Demonstration Interface [10.808201018448274]
  We propose feasibility-aware behavior cloning from observation (FABCO)<n>In the FABCO framework, the feasibility of each demonstration is assessed using the robot's pre-trained forward and inverse dynamics models.<n>We experimentally validated FABCO's effectiveness by applying it to a pipette insertion task involving a pipette and a vial.
  arXiv  Detail & Related papers  (2025-03-12T03:14:04Z)
• In-Context Imitation Learning via Next-Token Prediction [25.63412917032012]
  In-Context Robot Transformer (ICRT) performs autoregressive prediction on sensorimotor trajectories without relying on linguistic data or reward function. ICRT can adapt to new tasks specified by prompts, even in environment configurations that differ from both the prompt and the training data.
  arXiv  Detail & Related papers  (2024-08-28T17:50:19Z)
• Learning from Imperfect Demonstrations with Self-Supervision for Robotic Manipulation [30.791222277450053]
  Current imitation learning (IL) typically discards imperfect data, focusing solely on successful expert data.<n>We introduce a Self-Supervised Data Filtering framework (SSDF) that combines expert and imperfect data to compute quality scores for failed trajectory segments.<n>SSDF can accurately expand the training dataset with high-quality imperfect data and improve the success rates for all robotic manipulation tasks.
  arXiv  Detail & Related papers  (2024-01-17T04:15:56Z)
• Learning Video-Conditioned Policies for Unseen Manipulation Tasks [83.2240629060453]
  Video-conditioned Policy learning maps human demonstrations of previously unseen tasks to robot manipulation skills. We learn our policy to generate appropriate actions given current scene observations and a video of the target task. We validate our approach on a set of challenging multi-task robot manipulation environments and outperform state of the art.
  arXiv  Detail & Related papers  (2023-05-10T16:25:42Z)
• Learning Reward Functions for Robotic Manipulation by Observing Humans [92.30657414416527]
  We use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies. The learned rewards are based on distances to a goal in an embedding space learned using a time-contrastive objective.
  arXiv  Detail & Related papers  (2022-11-16T16:26:48Z)
• Instruction-driven history-aware policies for robotic manipulations [82.25511767738224]
  We propose a unified transformer-based approach that takes into account multiple inputs. In particular, our transformer architecture integrates (i) natural language instructions and (ii) multi-view scene observations. We evaluate our method on the challenging RLBench benchmark and on a real-world robot.
  arXiv  Detail & Related papers  (2022-09-11T16:28:25Z)
• Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation [55.31301153979621]
  We tackle real-world long-horizon robot manipulation tasks through skill discovery. We present a bottom-up approach to learning a library of reusable skills from unsegmented demonstrations. Our method has shown superior performance over state-of-the-art imitation learning methods in multi-stage manipulation tasks.
  arXiv  Detail & Related papers  (2021-09-28T16:18:54Z)
• Visual Imitation Made Easy [102.36509665008732]
  We present an alternate interface for imitation that simplifies the data collection process while allowing for easy transfer to robots. We use commercially available reacher-grabber assistive tools both as a data collection device and as the robot's end-effector. We experimentally evaluate on two challenging tasks: non-prehensile pushing and prehensile stacking, with 1000 diverse demonstrations for each task.
  arXiv  Detail & Related papers  (2020-08-11T17:58:50Z)
• Interactive Robot Training for Non-Markov Tasks [6.252236971703546]
  We propose a Bayesian interactive robot training framework that allows the robot to learn from both demonstrations provided by a teacher. We also present an active learning approach to identify the task execution with the most uncertain degree of acceptability. We demonstrate the efficacy of our approach in a real-world setting through a user-study based on teaching a robot to set a dinner table.
  arXiv  Detail & Related papers  (2020-03-04T18:19:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.