Spotlighting Task-Relevant Features: Object-Centric Representations for Better Generalization in Robotic Manipulation

• URL: http://arxiv.org/abs/2601.21416v1
• Date: Thu, 29 Jan 2026 08:55:53 GMT
• Title: Spotlighting Task-Relevant Features: Object-Centric Representations for Better Generalization in Robotic Manipulation
• Abstract summary: Slot-Based Object-Centric Representations (SBOCR) can group dense features into a finite set of object-like entities.<n>SBOCR is a promising direction for designing visual systems that generalize effectively in dynamic, real-world robotic environments.
• Score: 41.66477383019748
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The generalization capabilities of robotic manipulation policies are heavily influenced by the choice of visual representations. Existing approaches typically rely on representations extracted from pre-trained encoders, using two dominant types of features: global features, which summarize an entire image via a single pooled vector, and dense features, which preserve a patch-wise embedding from the final encoder layer. While widely used, both feature types mix task-relevant and irrelevant information, leading to poor generalization under distribution shifts, such as changes in lighting, textures, or the presence of distractors. In this work, we explore an intermediate structured alternative: Slot-Based Object-Centric Representations (SBOCR), which group dense features into a finite set of object-like entities. This representation permits to naturally reduce the noise provided to the robotic manipulation policy while keeping enough information to efficiently perform the task. We benchmark a range of global and dense representations against intermediate slot-based representations, across a suite of simulated and real-world manipulation tasks ranging from simple to complex. We evaluate their generalization under diverse visual conditions, including changes in lighting, texture, and the presence of distractors. Our findings reveal that SBOCR-based policies outperform dense and global representation-based policies in generalization settings, even without task-specific pretraining. These insights suggest that SBOCR is a promising direction for designing visual systems that generalize effectively in dynamic, real-world robotic environments.

Related papers

• Capturing Visual Environment Structure Correlates with Control Performance [62.69063991419694]
  Existing proxy metrics focus on the representation's capacity to capture narrow aspects of the visual world.<n>We measure how well pretrained visual encoders support decoding of environment state.<n>We show that this probing accuracy strongly correlates with downstream policy performance.
  arXiv  Detail & Related papers  (2026-02-04T18:59:12Z)
• STORM: Slot-based Task-aware Object-centric Representation for robotic Manipulation [44.055446967139424]
  STORM is a lightweight object-centric adaptation module that augments frozen visual foundation models with semantic-aware slots for robotic manipulation.<n>Storm employs a multi-phase training strategy: object-centric slots are stabilized through visual--semantic pretraining using language embeddings, then jointly adapted with a downstream manipulation policy.<n>Experiments show that STORM improves generalization to visual distractors, and control performance compared to directly using frozen foundation model features or training object-centric representations end-to-end.
  arXiv  Detail & Related papers  (2026-01-28T08:46:04Z)
• T-Rex: Task-Adaptive Spatial Representation Extraction for Robotic Manipulation with Vision-Language Models [35.83717913117858]
  We introduce T-Rex, a Task-Adaptive Framework for Spatial Representation Extraction.<n>We show that our approach delivers significant advantages in spatial understanding, efficiency, and stability without additional training.
  arXiv  Detail & Related papers  (2025-06-24T10:36:15Z)
• Zero-Shot Visual Generalization in Robot Manipulation [0.13280779791485384]
  Current approaches often sidestep the problem by relying on invariant representations such as point clouds and depth.<n>Disentangled representation learning has recently shown promise in enabling vision-based reinforcement learning policies to be robust to visual distribution shifts.<n>We demonstrate zero-shot adaptability to visual perturbations in both simulation and on real hardware.
  arXiv  Detail & Related papers  (2025-05-16T22:01:46Z)
• Object-Centric Representations Improve Policy Generalization in Robot Manipulation [43.18545365968973]
  We investigate object-centric representations (OCR) as a structured alternative that segments visual input into a finished set of entities.<n>We benchmark a range of visual encoders-object-centric, global and dense methods-across a suite of simulated and real-world manipulation tasks.<n>Our findings reveal that OCR-based policies outperform dense and global representations in generalization settings, even without task-specific pretraining.
  arXiv  Detail & Related papers  (2025-05-16T07:06:37Z)
• DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping [18.410329897882658]
  A general-purpose robot must be capable of grasping diverse objects in arbitrary scenarios.<n>We present DexGraspVLA, a hierarchical framework for robust generalization in language-guided general dexterous grasping.
  arXiv  Detail & Related papers  (2025-02-28T09:57:20Z)
• Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies.<n>Our findings are synthesized in Flex (Fly lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors.<n>We demonstrate the effectiveness of this approach on a quadrotor fly-to-target task, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• Neural Clustering based Visual Representation Learning [61.72646814537163]
  Clustering is one of the most classic approaches in machine learning and data analysis. We propose feature extraction with clustering (FEC), which views feature extraction as a process of selecting representatives from data. FEC alternates between grouping pixels into individual clusters to abstract representatives and updating the deep features of pixels with current representatives.
  arXiv  Detail & Related papers  (2024-03-26T06:04:50Z)
• What Makes Pre-Trained Visual Representations Successful for Robust Manipulation? [57.92924256181857]
  We find that visual representations designed for manipulation and control tasks do not necessarily generalize under subtle changes in lighting and scene texture. We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models.
  arXiv  Detail & Related papers  (2023-11-03T18:09:08Z)
• Vision Transformers: From Semantic Segmentation to Dense Prediction [139.15562023284187]
  We explore the global context learning potentials of vision transformers (ViTs) for dense visual prediction. Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information. We formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture.
  arXiv  Detail & Related papers  (2022-07-19T15:49:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.