Conditional Visual Servoing for Multi-Step Tasks

• URL: http://arxiv.org/abs/2205.08441v1
• Date: Tue, 17 May 2022 15:34:54 GMT
• Title: Conditional Visual Servoing for Multi-Step Tasks
• Abstract summary: We propose a modular approach to extend visual servoing to scenarios with multiple demonstration sequences. We call this conditional servoing, as we choose the next demonstration conditioned on the observation of the robot. This method presents an appealing strategy to tackle multi-step problems.
• Score: 38.549573781429565
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Visual Servoing has been effectively used to move a robot into specific target locations or to track a recorded demonstration. It does not require manual programming, but it is typically limited to settings where one demonstration maps to one environment state. We propose a modular approach to extend visual servoing to scenarios with multiple demonstration sequences. We call this conditional servoing, as we choose the next demonstration conditioned on the observation of the robot. This method presents an appealing strategy to tackle multi-step problems, as individual demonstrations can be combined flexibly into a control policy. We propose different selection functions and compare them on a shape-sorting task in simulation. With the reprojection error yielding the best overall results, we implement this selection function on a real robot and show the efficacy of the proposed conditional servoing. For videos of our experiments, please check out our project page: https://lmb.informatik.uni-freiburg.de/projects/conditional_servoing/

Related papers

• SparTa: Sparse Graphical Task Models from a Handful of Demonstrations [18.00619071013106]
  We focus on inferring what the robot should achieve in the task, rather than how to do so.<n>We propose a demonstration segmentation and pooling approach that extracts a series of manipulation graphs and estimates distributions over object states.<n>We deploy the fitted models both in simulation and on a real robot, demonstrating that the resulting task representations support reliable execution across environments.
  arXiv  Detail & Related papers  (2026-02-18T21:54:35Z)
• Improving Generalization of Language-Conditioned Robot Manipulation [29.405161073483175]
  We present a framework that learns object-arrangement tasks from just a few demonstrations.<n>We validate our method on both simulation and real-world robotic environments.
  arXiv  Detail & Related papers  (2025-08-04T13:29:26Z)
• Screw Geometry Meets Bandits: Incremental Acquisition of Demonstrations to Generate Manipulation Plans [9.600625243282618]
  We study the problem of methodically obtaining a sufficient set of kinesthetic demonstrations, one at a time. We present a novel approach to address these open problems using (i) a screw geometric representation to generate manipulation plans from demonstrations. We present experimental results on two example manipulation tasks, namely, pouring and scooping, to illustrate our approach.
  arXiv  Detail & Related papers  (2024-10-23T20:57:56Z)
• PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs [140.14239499047977]
  Vision language models (VLMs) have shown impressive capabilities across a variety of tasks, from logical reasoning to visual understanding. We propose a novel visual prompting approach for VLMs that we call Prompting with Iterative Visual Optimization (PIVOT) We find, perhaps surprisingly, that our approach enables zero-shot control of robotic systems without any robot training data, navigation in a variety of environments, and other capabilities.
  arXiv  Detail & Related papers  (2024-02-12T18:33:47Z)
• Learning to Act from Actionless Videos through Dense Correspondences [87.1243107115642]
  We present an approach to construct a video-based robot policy capable of reliably executing diverse tasks across different robots and environments. Our method leverages images as a task-agnostic representation, encoding both the state and action information, and text as a general representation for specifying robot goals. We demonstrate the efficacy of our approach in learning policies on table-top manipulation and navigation tasks.
  arXiv  Detail & Related papers  (2023-10-12T17:59:23Z)
• Compositional Servoing by Recombining Demonstrations [32.23118166912325]
  We present a framework that formulates the visual servoing task as graph traversal. We construct demonstration graphs by splitting existing demonstrations and recombining them. We show that recombining demonstrations leads to higher task-respective success.
  arXiv  Detail & Related papers  (2023-10-06T14:16:49Z)
• RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation [36.43143326197769]
  Track-Any-Point (TAP) models isolate the relevant motion in a demonstration, and parameterize a low-level controller to reproduce this motion across changes in the scene configuration. We show this results in robust robot policies that can solve complex object-arrangement tasks such as shape-matching, stacking, and even full path-following tasks such as applying glue and sticking objects together.
  arXiv  Detail & Related papers  (2023-08-30T11:57:04Z)
• Program Generation from Diverse Video Demonstrations [49.202289347899836]
  Generalising over multiple observations is a task that has historically presented difficulties for machines to grasp. We propose a model that can extract general rules from video demonstrations by simultaneously performing summarisation and translation.
  arXiv  Detail & Related papers  (2023-02-01T01:51:45Z)
• VIMA: General Robot Manipulation with Multimodal Prompts [82.01214865117637]
  We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts. We develop a new simulation benchmark that consists of thousands of procedurally-generated tabletop tasks. We design a transformer-based robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively.
  arXiv  Detail & Related papers  (2022-10-06T17:50:11Z)
• Learning to Shift Attention for Motion Generation [55.61994201686024]
  One challenge of motion generation using robot learning from demonstration techniques is that human demonstrations follow a distribution with multiple modes for one task query. Previous approaches fail to capture all modes or tend to average modes of the demonstrations and thus generate invalid trajectories. We propose a motion generation model with extrapolation ability to overcome this problem.
  arXiv  Detail & Related papers  (2021-02-24T09:07:52Z)
• Robust Maximum Entropy Behavior Cloning [15.713997170792842]
  Imitation learning (IL) algorithms use expert demonstrations to learn a specific task. Most of the existing approaches assume that all expert demonstrations are reliable and trustworthy, but what if there exist some adversarial demonstrations among the given data-set? We propose a novel general frame-work to directly generate a policy from demonstrations that autonomously detect the adversarial demonstrations and exclude them from the data set.
  arXiv  Detail & Related papers  (2021-01-04T22:08:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.