Viewpoint Matters: Dynamically Optimizing Viewpoints with Masked Autoencoder for Visual Manipulation

• URL: http://arxiv.org/abs/2602.04243v2
• Date: Thu, 05 Mar 2026 14:34:26 GMT
• Title: Viewpoint Matters: Dynamically Optimizing Viewpoints with Masked Autoencoder for Visual Manipulation
• Abstract summary: We propose MAE-Select, a novel framework for active viewpoint selection in single-camera robotic systems.<n>MaE-Select fully leverages pre-trained multi-view masked autoencoder representations and dynamically selects the next most informative viewpoint at each time chunk.<n>Experiments demonstrate that MAE-Select improves the capabilities of single-camera systems and, in some cases, even surpasses multi-camera setups.
• Score: 9.420906356149874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robotic manipulation continues to be a challenge, and imitation learning (IL) enables robots to learn tasks from expert demonstrations. Current IL methods typically rely on fixed camera setups, where cameras are manually positioned in static locations, imposing significant limitations on adaptability and coverage. Inspired by human active perception, where humans dynamically adjust their viewpoint to capture the most relevant and least noisy information, we propose MAE-Select, a novel framework for active viewpoint selection in single-camera robotic systems. MAE-Select fully leverages pre-trained multi-view masked autoencoder representations and dynamically selects the next most informative viewpoint at each time chunk without requiring labeled viewpoints. Extensive experiments demonstrate that MAE-Select improves the capabilities of single-camera systems and, in some cases, even surpasses multi-camera setups. The project will be available at https://mae-select.github.io.

Related papers

• Multi-Camera View Scaling for Data-Efficient Robot Imitation Learning [57.88781687419521]
  Multiple synchronized camera perspectives are used to generate pseudo-demonstrations from each expert trajectory.<n>Experiments in simulation and real-world manipulation tasks demonstrate significant gains in data efficiency and generalization.<n>Our results suggest that scaling camera views provides a practical and scalable solution for imitation learning.
  arXiv  Detail & Related papers  (2026-04-01T07:00:44Z)
• ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos [15.809468471562537]
  ZeroMimic generates image goal-conditioned skill policies for several common manipulation tasks.<n>We evaluate ZeroMimic's out-of-the-box performance in varied real-world and simulated kitchen settings.<n>To enable plug-and-play reuse of ZeroMimic policies on other task setups and robots, we release software and policy checkpoints.
  arXiv  Detail & Related papers  (2025-03-31T09:27:00Z)
• Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos [66.1935609072708]
  LangView is a framework that uses the relative accuracy of view-dependent caption predictions as a proxy for best view pseudo-labels.<n>During inference, our model takes as input only a multi-view video--no language or camera poses--and returns the best viewpoint to watch at each timestep.
  arXiv  Detail & Related papers  (2024-11-13T16:31:08Z)
• PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs [140.14239499047977]
  Vision language models (VLMs) have shown impressive capabilities across a variety of tasks, from logical reasoning to visual understanding. We propose a novel visual prompting approach for VLMs that we call Prompting with Iterative Visual Optimization (PIVOT) We find, perhaps surprisingly, that our approach enables zero-shot control of robotic systems without any robot training data, navigation in a variety of environments, and other capabilities.
  arXiv  Detail & Related papers  (2024-02-12T18:33:47Z)
• Learning to Select Camera Views: Efficient Multiview Understanding at Few Glances [59.34619548026885]
  We propose a view selection approach that analyzes the target object or scenario from given views and selects the next best view for processing. Our approach features a reinforcement learning based camera selection module, MVSelect, that not only selects views but also facilitates joint training with the task network.
  arXiv  Detail & Related papers  (2023-03-10T18:59:10Z)
• Multi-View Masked World Models for Visual Robotic Manipulation [132.97980128530017]
  We train a multi-view masked autoencoder which reconstructs pixels of randomly masked viewpoints. We demonstrate the effectiveness of our method in a range of scenarios. We also show that the multi-view masked autoencoder trained with multiple randomized viewpoints enables training a policy with strong viewpoint randomization.
  arXiv  Detail & Related papers  (2023-02-05T15:37:02Z)
• SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation [101.55622133406446]
  We propose a SurroundDepth method to incorporate the information from multiple surrounding views to predict depth maps across cameras. Specifically, we employ a joint network to process all the surrounding views and propose a cross-view transformer to effectively fuse the information from multiple views. In experiments, our method achieves the state-of-the-art performance on the challenging multi-camera depth estimation datasets.
  arXiv  Detail & Related papers  (2022-04-07T17:58:47Z)
• Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulation [15.632809977544907]
  Learning to solve precision-based manipulation tasks from visual feedback could drastically reduce the engineering efforts required by traditional robot systems. We propose a setting for robotic manipulation in which the agent receives visual feedback from both a third-person camera and an egocentric camera mounted on the robot's wrist. To fuse visual information from both cameras effectively, we additionally propose to use Transformers with a cross-view attention mechanism.
  arXiv  Detail & Related papers  (2022-01-19T18:39:03Z)
• Multi-Sensor Next-Best-View Planning as Matroid-Constrained Submodular Maximization [42.2136160962607]
  We consider the problem of creating a 3D model using depth images captured by a team of multiple robots. Next-best-view planning uses the current scene model to select the next viewpoints. We show that multi-sensor next-best-view planning with this utility function is an instance of submodular under a matroid constraint.
  arXiv  Detail & Related papers  (2020-07-04T12:28:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.