Capturing Visual Environment Structure Correlates with Control Performance

• URL: http://arxiv.org/abs/2602.04880v1
• Date: Wed, 04 Feb 2026 18:59:12 GMT
• Title: Capturing Visual Environment Structure Correlates with Control Performance
• Abstract summary: Existing proxy metrics focus on the representation's capacity to capture narrow aspects of the visual world.<n>We measure how well pretrained visual encoders support decoding of environment state.<n>We show that this probing accuracy strongly correlates with downstream policy performance.
• Score: 62.69063991419694
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The choice of visual representation is key to scaling generalist robot policies. However, direct evaluation via policy rollouts is expensive, even in simulation. Existing proxy metrics focus on the representation's capacity to capture narrow aspects of the visual world, like object shape, limiting generalization across environments. In this paper, we take an analytical perspective: we probe pretrained visual encoders by measuring how well they support decoding of environment state -- including geometry, object structure, and physical attributes -- from images. Leveraging simulation environments with access to ground-truth state, we show that this probing accuracy strongly correlates with downstream policy performance across diverse environments and learning settings, significantly outperforming prior metrics and enabling efficient representation selection. More broadly, our study provides insight into the representational properties that support generalizable manipulation, suggesting that learning to encode the latent physical state of the environment is a promising objective for control.

Related papers

• I-Perceive: A Foundation Model for Active Perception with Language Instructions [41.67607728608853]
  We propose I-Perceive, a foundation model for active perception conditioned on natural language instructions.<n>I-Perceive predicts camera views that follow open-ended language instructions, based on image-based scene contexts.<n> Experiments demonstrate that I-Perceive significantly outperforms state-of-the-art VLMs in both prediction accuracy and instruction following of generated camera views.
  arXiv  Detail & Related papers  (2026-02-28T11:38:56Z)
• Spotlighting Task-Relevant Features: Object-Centric Representations for Better Generalization in Robotic Manipulation [41.66477383019748]
  Slot-Based Object-Centric Representations (SBOCR) can group dense features into a finite set of object-like entities.<n>SBOCR is a promising direction for designing visual systems that generalize effectively in dynamic, real-world robotic environments.
  arXiv  Detail & Related papers  (2026-01-29T08:55:53Z)
• Object-Centric Representations Improve Policy Generalization in Robot Manipulation [43.18545365968973]
  We investigate object-centric representations (OCR) as a structured alternative that segments visual input into a finished set of entities.<n>We benchmark a range of visual encoders-object-centric, global and dense methods-across a suite of simulated and real-world manipulation tasks.<n>Our findings reveal that OCR-based policies outperform dense and global representations in generalization settings, even without task-specific pretraining.
  arXiv  Detail & Related papers  (2025-05-16T07:06:37Z)
• Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features [59.892436892964376]
  We investigate the minimal data requirements and architectural adaptations necessary to achieve robust closed-loop performance with vision-based control policies.<n>Our findings are synthesized in Flex (Fly lexically), a framework that uses pre-trained Vision Language Models (VLMs) as frozen patch-wise feature extractors.<n>We demonstrate the effectiveness of this approach on a quadrotor fly-to-target task, where agents trained via behavior cloning successfully generalize to real-world scenes.
  arXiv  Detail & Related papers  (2024-10-16T19:59:31Z)
• GAN Based Top-Down View Synthesis in Reinforcement Learning Environments [0.36427222462368697]
  This project explores learning the top-down view of an RL environment based on the artificial agent's first-person view observations with a generative adversarial network(GAN) The focus of the project is to learn the top-down view of an RL environment. It doesn't deal with any Reinforcement Learning task.
  arXiv  Detail & Related papers  (2024-10-16T08:44:23Z)
• Zero-Shot Object-Centric Representation Learning [72.43369950684057]
  We study current object-centric methods through the lens of zero-shot generalization. We introduce a benchmark comprising eight different synthetic and real-world datasets. We find that training on diverse real-world images improves transferability to unseen scenarios.
  arXiv  Detail & Related papers  (2024-08-17T10:37:07Z)
• What Makes Pre-Trained Visual Representations Successful for Robust Manipulation? [57.92924256181857]
  We find that visual representations designed for manipulation and control tasks do not necessarily generalize under subtle changes in lighting and scene texture. We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models.
  arXiv  Detail & Related papers  (2023-11-03T18:09:08Z)
• Learning Generalizable Manipulation Policies with Object-Centric 3D Representations [65.55352131167213]
  GROOT is an imitation learning method for learning robust policies with object-centric and 3D priors. It builds policies that generalize beyond their initial training conditions for vision-based manipulation. GROOT's performance excels in generalization over background changes, camera viewpoint shifts, and the presence of new object instances.
  arXiv  Detail & Related papers  (2023-10-22T18:51:45Z)
• Policy Gradient Methods in the Presence of Symmetries and State Abstractions [46.66541516203923]
  Reinforcement learning (RL) on high-dimensional and complex problems relies on abstraction for improved efficiency and generalization. We study abstraction in the continuous-control setting, and extend the definition of Markov decision process (MDP) homomorphisms to the setting of continuous state and action spaces. We propose a family of actor-critic algorithms that are able to learn the policy and the MDP homomorphism map simultaneously.
  arXiv  Detail & Related papers  (2023-05-09T17:59:10Z)
• SceneGen: Generative Contextual Scene Augmentation using Scene Graph Priors [3.1969855247377827]
  We introduce SceneGen, a generative contextual augmentation framework that predicts virtual object positions and orientations within existing scenes. SceneGen takes a semantically segmented scene as input, and outputs positional and orientational probability maps for placing virtual content. We formulate a novel spatial Scene Graph representation, which encapsulates explicit topological properties between objects, object groups, and the room. To demonstrate our system in action, we develop an Augmented Reality application, in which objects can be contextually augmented in real-time.
  arXiv  Detail & Related papers  (2020-09-25T18:36:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.