Choose What to Observe: Task-Aware Semantic-Geometric Representations for Visuomotor Policy

• URL: http://arxiv.org/abs/2603.07875v1
• Date: Mon, 09 Mar 2026 01:21:26 GMT
• Title: Choose What to Observe: Task-Aware Semantic-Geometric Representations for Visuomotor Policy
• Abstract summary: Visuomotor policies learn from demonstrations often overfit to nuisance visual factors in raw RGB observations.<n>We propose a task-aware observation interface that canonicalizes visual input into a shared representation.<n>We evaluate on RoboMimic (Lift), ManiSkill YCB grasping under clutter, four RLBench tasks under controlled appearance shifts, and two real-world Franka tasks.
• Score: 66.60668908340429
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visuomotor policies learned from demonstrations often overfit to nuisance visual factors in raw RGB observations, resulting in brittle behavior under appearance shifts such as background changes and object recoloring. We propose a task-aware observation interface that canonicalizes visual input into a shared representation, improving robustness to out-of-distribution (OOD) appearance changes without modifying or fine-tuning the policy. Given an RGB image and an open-vocabulary specification of task-relevant entities, we use SAM3 to segment the target object and robot/gripper. We construct an L0 observation by repainting segmented entities with predefined semantic colors on a constant background. For tasks requiring stronger geometric cues, we further inject monocular depth from Depth Anything 3 into the segmented regions via depth-guided overwrite, yielding a unified semantic--geometric observation (L1) that remains a standard 3-channel, image-like input. We evaluate on RoboMimic (Lift), ManiSkill YCB grasping under clutter, four RLBench tasks under controlled appearance shifts, and two real-world Franka tasks (ReachX and CloseCabinet). Across benchmarks and policy backbones (Flow Matching Policy and SmolVLA), our interface preserves in-distribution performance while substantially improving robustness under OOD visual shifts.

Related papers

• DINOcular: Self-Supervised Visuospatial Representations [7.360872501623849]
  We introduce a self-supervised framework for learning joint visuospatial representations from RGB-D observations.<n>Our method integrates depth-derived geometric priors with a visual backbone through inter-patch and intra-patch fusion.<n>The resulting representation shows promising improvements on 3D awareness while preserving semantic transfer.
  arXiv  Detail & Related papers  (2026-08-27T15:09:32Z)
• 3SGen: Unified Subject, Style, and Structure-Driven Image Generation with Adaptive Task-specific Memory [54.056509629389915]
  3SGen is a task-aware unified framework that performs all three conditioning modes within a single model.<n>At its core, an Adaptive Task-specific Memory (ATM) module dynamically disentangles, stores, and retrieves condition-specific priors.<n>We propose 3SGen-Bench, a unified image-driven generation benchmark with standardized metrics for evaluating cross-task fidelity and controllability.
  arXiv  Detail & Related papers  (2025-12-22T11:07:27Z)
• Correspondence-Oriented Imitation Learning: Flexible Visuomotor Control with 3D Conditioning [11.847696426774732]
  We introduce Imitation-Oriented Learning (COIL), a conditional policy learning framework for visuomotor control with a flexible task representation in 3D.<n>COIL generalizes across tasks, objects, and motion patterns, achieving superior granularity compared to prior methods on real-world manipulation tasks under both sparse and dense specifications.
  arXiv  Detail & Related papers  (2025-12-05T18:50:17Z)
• Learning Generalizable Manipulation Policies with Object-Centric 3D Representations [65.55352131167213]
  GROOT is an imitation learning method for learning robust policies with object-centric and 3D priors. It builds policies that generalize beyond their initial training conditions for vision-based manipulation. GROOT's performance excels in generalization over background changes, camera viewpoint shifts, and the presence of new object instances.
  arXiv  Detail & Related papers  (2023-10-22T18:51:45Z)
• Evaluating Robustness of Visual Representations for Object Assembly Task Requiring Spatio-Geometrical Reasoning [8.626019848533707]
  This paper focuses on evaluating and benchmarking the robustness of visual representations in the context of object assembly tasks. We employ a general framework in visuomotor policy learning that utilizes visual pretraining models as vision encoders. Our study investigates the robustness of this framework when applied to a dual-arm manipulation setup, specifically to the grasp variations.
  arXiv  Detail & Related papers  (2023-10-15T20:41:07Z)
• Self-Supervised Monocular Depth Estimation by Direction-aware Cumulative Convolution Network [80.19054069988559]
  We find that self-supervised monocular depth estimation shows a direction sensitivity and environmental dependency. We propose a new Direction-aware Cumulative Convolution Network (DaCCN), which improves the depth representation in two aspects. Experiments show that our method achieves significant improvements on three widely used benchmarks.
  arXiv  Detail & Related papers  (2023-08-10T14:32:18Z)
• Point-GCC: Universal Self-supervised 3D Scene Pre-training via Geometry-Color Contrast [9.14535402695962]
  Geometry and color information provided by point clouds are crucial for 3D scene understanding. We propose a universal 3D scene pre-training framework via Geometry-Color Contrast (Point-GCC) Point-GCC aligns geometry and color information using a Siamese network.
  arXiv  Detail & Related papers  (2023-05-31T07:44:03Z)
• CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion [20.121597331207276]
  Masked Image Modeling (MIM) has recently been established as a potent pre-training paradigm. In this paper we seek to learn representations that transfer well to a wide variety of 3D vision and lower-level geometric downstream tasks. Our experiments show that our pretext task leads to significantly improved performance for monocular 3D vision downstream tasks.
  arXiv  Detail & Related papers  (2022-10-19T16:50:36Z)
• Multitask AET with Orthogonal Tangent Regularity for Dark Object Detection [84.52197307286681]
  We propose a novel multitask auto encoding transformation (MAET) model to enhance object detection in a dark environment. In a self-supervision manner, the MAET learns the intrinsic visual structure by encoding and decoding the realistic illumination-degrading transformation. We have achieved the state-of-the-art performance using synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-05-06T16:27:14Z)
• Appearance Consensus Driven Self-Supervised Human Mesh Recovery [67.20942777949793]
  We present a self-supervised human mesh recovery framework to infer human pose and shape from monocular images. We achieve state-of-the-art results on the standard model-based 3D pose estimation benchmarks. The resulting colored mesh prediction opens up the usage of our framework for a variety of appearance-related tasks beyond the pose and shape estimation.
  arXiv  Detail & Related papers  (2020-08-04T05:40:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.