ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representations

• URL: http://arxiv.org/abs/2509.11125v1
• Date: Sun, 14 Sep 2025 06:31:04 GMT
• Title: ManiVID-3D: Generalizable View-Invariant Reinforcement Learning for Robotic Manipulation via Disentangled 3D Representations
• Authors: Zheng Li, Pei Qu, Yufei Jia, Shihui Zhou, Haizhou Ge, Jiahang Cao, Jinni Zhou, Guyue Zhou, Jun Ma,
• Abstract summary: ManiVID-3D is a novel 3D visual reinforcement learning architecture for robotic manipulation.<n>It learns view-invariant representations through self-supervised disentangled feature learning.<n>It achieves a 44.7% higher success rate than state-of-the-art methods under viewpoint variations.
• Score: 19.02933938928656
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deploying visual reinforcement learning (RL) policies in real-world manipulation is often hindered by camera viewpoint changes. A policy trained from a fixed front-facing camera may fail when the camera is shifted--an unavoidable situation in real-world settings where sensor placement is hard to manage appropriately. Existing methods often rely on precise camera calibration or struggle with large perspective changes. To address these limitations, we propose ManiVID-3D, a novel 3D RL architecture designed for robotic manipulation, which learns view-invariant representations through self-supervised disentangled feature learning. The framework incorporates ViewNet, a lightweight yet effective module that automatically aligns point cloud observations from arbitrary viewpoints into a unified spatial coordinate system without the need for extrinsic calibration. Additionally, we develop an efficient GPU-accelerated batch rendering module capable of processing over 5000 frames per second, enabling large-scale training for 3D visual RL at unprecedented speeds. Extensive evaluation across 10 simulated and 5 real-world tasks demonstrates that our approach achieves a 44.7% higher success rate than state-of-the-art methods under viewpoint variations while using 80% fewer parameters. The system's robustness to severe perspective changes and strong sim-to-real performance highlight the effectiveness of learning geometrically consistent representations for scalable robotic manipulation in unstructured environments. Our project website can be found in https://zheng-joe-lee.github.io/manivid3d/.

Related papers

• PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning [5.308328605042682]
  360 panoramic images are increasingly used in virtual reality, autonomous driving, and robotics for holistic scene understanding.<n>Current Vision-Language Models (VLMs) struggle with 3D spatial reasoning on Equirectangular Projection (ERP) images due to geometric distortion and limited 3D supervision.<n>We introduce PanoEnv, a large-scale VQA benchmark built from synthetic 3D environments.<n>Our 7B model achieves new state-of-the-art performance, improving overall accuracy to 52.93% (+3.59%) and open-ended accuracy to 14.83% while maintaining structured-task performance.
  arXiv  Detail & Related papers  (2026-02-25T15:12:17Z)
• Learning Humanoid End-Effector Control for Open-Vocabulary Visual Loco-Manipulation [14.013652439013692]
  This paper presents a new paradigm, HERO, for object loco-manipulation with humanoid robots.<n>We achieve this by designing an accurate residual-aware EE tracking policy.<n>We use this accurate end-effector tracker to build a modular system for loco-manipulation.
  arXiv  Detail & Related papers  (2026-02-18T18:55:02Z)
• Think3D: Thinking with Space for Spatial Reasoning [54.518667686880114]
  We introduce Think3D, a framework that enables vision large models (VLMs) to think with 3D space.<n>Without additional training, Think3D significantly improves the spatial reasoning performance of advanced models.<n>Our findings demonstrate that training-free, tool-augmented spatial exploration is a viable path toward more flexible and human-like 3D reasoning in multimodal agents.
  arXiv  Detail & Related papers  (2026-01-19T13:13:54Z)
• Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots [55.43376513158555]
  Camera Depth Models (CDMs) are a simple plugin on daily-use depth cameras.<n>We develop a neural data engine that generates high-quality paired data from simulation by modeling a depth camera's noise pattern.<n>For the first time, our experiments demonstrate, for the first time, that a policy trained on raw simulated depth, without the need for adding noise or real-world fine-tuning, generalizes seamlessly to real-world robots.
  arXiv  Detail & Related papers  (2025-09-02T17:29:38Z)
• CL3R: 3D Reconstruction and Contrastive Learning for Enhanced Robotic Manipulation Representations [19.71090711790973]
  We propose a novel 3D pre-training framework designed to enhance robotic manipulation policies.<n>Our method integrates both spatial awareness and semantic understanding by employing a point cloud Masked Autoencoder.<n>We mitigate camera view ambiguity and improve generalization, enabling robust perception from novel viewpoints at test time.
  arXiv  Detail & Related papers  (2025-07-11T02:16:32Z)
• Easi3R: Estimating Disentangled Motion from DUSt3R Without Training [69.51086319339662]
  We introduce Easi3R, a simple yet efficient training-free method for 4D reconstruction.<n>Our approach applies attention adaptation during inference, eliminating the need for from-scratch pre-training or network fine-tuning.<n>Our experiments on real-world dynamic videos demonstrate that our lightweight attention adaptation significantly outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-03-31T17:59:58Z)
• Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos [76.07894127235058]
  We present a system for mining high-quality 4D reconstructions from internet stereoscopic, wide-angle videos.<n>We use this method to generate large-scale data in the form of world-consistent, pseudo-metric 3D point clouds.<n>We demonstrate the utility of this data by training a variant of DUSt3R to predict structure and 3D motion from real-world image pairs.
  arXiv  Detail & Related papers  (2024-12-12T18:59:54Z)
• Part-Guided 3D RL for Sim2Real Articulated Object Manipulation [27.422878372169805]
  We propose a part-guided 3D RL framework, which can learn to manipulate articulated objects without demonstrations. We combine the strengths of 2D segmentation and 3D RL to improve the efficiency of RL policy training. A single versatile RL policy can be trained on multiple articulated object manipulation tasks simultaneously in simulation.
  arXiv  Detail & Related papers  (2024-04-26T10:18:17Z)
• PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm [111.16358607889609]
  We introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation.<n>For the first time, PonderV2 achieves state-of-the-art performance on 11 indoor and outdoor benchmarks, implying its effectiveness.
  arXiv  Detail & Related papers  (2023-10-12T17:59:57Z)
• Visual Reinforcement Learning with Self-Supervised 3D Representations [15.991546692872841]
  We present a unified framework for self-supervised learning of 3D representations for motor control. Our method enjoys improved sample efficiency in simulated manipulation tasks compared to 2D representation learning methods.
  arXiv  Detail & Related papers  (2022-10-13T17:59:55Z)
• Unsupervised Learning of Visual 3D Keypoints for Control [104.92063943162896]
  Learning sensorimotor control policies from high-dimensional images crucially relies on the quality of the underlying visual representations. We propose a framework to learn such a 3D geometric structure directly from images in an end-to-end unsupervised manner. These discovered 3D keypoints tend to meaningfully capture robot joints as well as object movements in a consistent manner across both time and 3D space.
  arXiv  Detail & Related papers  (2021-06-14T17:59:59Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)
• CRAVES: Controlling Robotic Arm with a Vision-based Economic System [96.56564257199474]
  Training a robotic arm to accomplish real-world tasks has been attracting increasing attention in both academia and industry.<n>This work discusses the role of computer vision algorithms in this field.<n>We present an alternative solution, which uses a 3D model to create a large number of synthetic data.
  arXiv  Detail & Related papers  (2018-12-03T13:28:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.