Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning

• URL: http://arxiv.org/abs/2512.05953v1
• Date: Fri, 05 Dec 2025 18:50:17 GMT
• Title: Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning
• Abstract summary: We introduce Imitation-Oriented Learning (COIL), a conditional policy learning framework for visuomotor control with a flexible task representation in 3D.<n>COIL generalizes across tasks, objects, and motion patterns, achieving superior granularity compared to prior methods on real-world manipulation tasks under both sparse and dense specifications.
• Score: 11.847696426774732
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Correspondence-Oriented Imitation Learning (COIL), a conditional policy learning framework for visuomotor control with a flexible task representation in 3D. At the core of our approach, each task is defined by the intended motion of keypoints selected on objects in the scene. Instead of assuming a fixed number of keypoints or uniformly spaced time intervals, COIL supports task specifications with variable spatial and temporal granularity, adapting to different user intents and task requirements. To robustly ground this correspondence-oriented task representation into actions, we design a conditional policy with a spatio-temporal attention mechanism that effectively fuses information across multiple input modalities. The policy is trained via a scalable self-supervised pipeline using demonstrations collected in simulation, with correspondence labels automatically generated in hindsight. COIL generalizes across tasks, objects, and motion patterns, achieving superior performance compared to prior methods on real-world manipulation tasks under both sparse and dense specifications.

Related papers

• `Attention-Guided Cross-Temporal Clustering for Self-Supervised Video Object Segmentation [16.700937089524803]
  Video object segmentation (VOS) is a fundamental task in video understanding, requiring accurate delineation and consistent tracking of objects across frames.<n>We propose a self-supervised framework, Cross-Temporal Consistency and Clustering, that learns part-aware representations by combining attention-guided token selection with lightweight temporal clustering.
  arXiv  Detail & Related papers  (2026-07-08T10:13:20Z)
• Task Editing for Generalizable 3D Visuomotor Policy Learning [50.590696755853365]
  3D visuomotor policies offer a promising direction for complex robotic manipulation, as depth maps and point clouds provide rich geometric information for spatial reasoning.<n>Existing methods commonly use demonstration generation strategies to improve data efficiency by applying object-centric transformations to human-collected demonstrations.<n>We propose Task-Edit, a novel demonstration generation framework that generates diverse trajectories from a task-centric editing perspective.
  arXiv  Detail & Related papers  (2026-06-05T07:54:49Z)
• Choose What to Observe: Task-Aware Semantic-Geometric Representations for Visuomotor Policy [66.60668908340429]
  Visuomotor policies learn from demonstrations often overfit to nuisance visual factors in raw RGB observations.<n>We propose a task-aware observation interface that canonicalizes visual input into a shared representation.<n>We evaluate on RoboMimic (Lift), ManiSkill YCB grasping under clutter, four RLBench tasks under controlled appearance shifts, and two real-world Franka tasks.
  arXiv  Detail & Related papers  (2026-03-09T01:21:26Z)
• Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents [52.30603055218294]
  Trajectory2Task is a verifiable data generation pipeline for studying tool use at scale under three realistic user scenarios.<n>It converts valid tool-call trajectories into user-facing tasks with controlled intent adaptations.<n>We benchmark seven state-of-the-art LLMs on the generated complex user scenario tasks and observe frequent failures.
  arXiv  Detail & Related papers  (2026-01-28T00:36:13Z)
• Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies.<n>Our findings are synthesized in Flex (Fly lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors.<n>We demonstrate the effectiveness of this approach on a quadrotor fly-to-target task, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• MATCH POLICY: A Simple Pipeline from Point Cloud Registration to Manipulation Policies [25.512068008948603]
  MATCH POLICY is a pipeline for solving high-precision pick and place tasks.<n>It transfers action inference into a point cloud registration task.<n>It achieves extremely high sample efficiency and generalizability to unseen configurations.
  arXiv  Detail & Related papers  (2024-09-23T20:09:43Z)
• RepVF: A Unified Vector Fields Representation for Multi-task 3D Perception [64.80760846124858]
  This paper proposes a novel unified representation, RepVF, which harmonizes the representation of various perception tasks. RepVF characterizes the structure of different targets in the scene through a vector field, enabling a single-head, multi-task learning model. Building upon RepVF, we introduce RFTR, a network designed to exploit the inherent connections between different tasks.
  arXiv  Detail & Related papers  (2024-07-15T16:25:07Z)
• Task-conditioned adaptation of visual features in multi-task policy learning [9.320904829966588]
  We introduce task-conditioned adapters that do not require finetuning any pre-trained weights, combined with a single policy trained with behavior cloning. We evaluate the method on a wide variety of tasks from the CortexBench benchmark and show that, compared to existing work, it can be addressed with a single policy.
  arXiv  Detail & Related papers  (2024-02-12T15:57:31Z)
• A Dynamic Feature Interaction Framework for Multi-task Visual Perception [100.98434079696268]
  We devise an efficient unified framework to solve multiple common perception tasks. These tasks include instance segmentation, semantic segmentation, monocular 3D detection, and depth estimation. Our proposed framework, termed D2BNet, demonstrates a unique approach to parameter-efficient predictions for multi-task perception.
  arXiv  Detail & Related papers  (2023-06-08T09:24:46Z)
• Taskology: Utilizing Task Relations at Scale [28.09712466727001]
  We show that we can leverage the inherent relationships among collections of tasks, as they are trained jointly. explicitly utilizing the relationships between tasks allows improving their performance while dramatically reducing the need for labeled data. We demonstrate our framework on subsets of the following collection of tasks: depth and normal prediction, semantic segmentation, 3D motion and ego-motion estimation, and object tracking and 3D detection in point clouds.
  arXiv  Detail & Related papers  (2020-05-14T22:53:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.