RoboTAG: End-to-end Robot Configuration Estimation via Topological Alignment Graph

• URL: http://arxiv.org/abs/2511.07717v1
• Date: Wed, 12 Nov 2025 01:12:40 GMT
• Title: RoboTAG: End-to-end Robot Configuration Estimation via Topological Alignment Graph
• Authors: Yifan Liu, Fangneng Zhan, Wanhua Li, Haowen Sun, Katerina Fragkiadaki, Hanspeter Pfister,
• Abstract summary: Estimating robot pose from a monocular RGB image is a challenge in robotics and computer vision.<n>Existing methods typically build networks on top of 2D visual backbones and depend heavily on labeled data for training.<n>We propose Robot Topological Alignment Graph (RoboTAG), which incorporates a 3D branch to inject 3D priors while enabling co-evolution of the 2D and 3D representations.
• Score: 62.270763554624615
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Estimating robot pose from a monocular RGB image is a challenge in robotics and computer vision. Existing methods typically build networks on top of 2D visual backbones and depend heavily on labeled data for training, which is often scarce in real-world scenarios, causing a sim-to-real gap. Moreover, these approaches reduce the 3D-based problem to 2D domain, neglecting the 3D priors. To address these, we propose Robot Topological Alignment Graph (RoboTAG), which incorporates a 3D branch to inject 3D priors while enabling co-evolution of the 2D and 3D representations, alleviating the reliance on labels. Specifically, the RoboTAG consists of a 3D branch and a 2D branch, where nodes represent the states of the camera and robot system, and edges capture the dependencies between these variables or denote alignments between them. Closed loops are then defined in the graph, on which a consistency supervision across branches can be applied. This design allows us to utilize in-the-wild images as training data without annotations. Experimental results demonstrate that our method is effective across robot types, highlighting its potential to alleviate the data bottleneck in robotics.

Related papers

• RoboEye: Enhancing 2D Robotic Object Identification with Selective 3D Geometric Keypoint Matching [5.240139281459202]
  RoboEye is a framework that augments 2D semantic features with domain-adapted 3D reasoning and lightweight adapters.<n> Experiments show that RoboEye improves Recall@1 by 7.1% over the prior state of the art.
  arXiv  Detail & Related papers  (2025-09-18T13:59:24Z)
• Pixels-to-Graph: Real-time Integration of Building Information Models and Scene Graphs for Semantic-Geometric Human-Robot Understanding [6.924983239916623]
  We introduce Pixels-to-Graph (Pix2G), a novel lightweight method to generate structured scene graphs from image pixels and LiDAR maps in real-time.<n>The framework is designed to perform all operation on CPU only to satisfy onboard compute constraints.<n>The proposed method is quantitatively and qualitatively evaluated during real-world experiments performed using the NASA JPL NeBula-Spot legged robot.
  arXiv  Detail & Related papers  (2025-06-27T19:23:31Z)
• Unifying 2D and 3D Vision-Language Understanding [85.84054120018625]
  We introduce UniVLG, a unified architecture for 2D and 3D vision-language learning.<n>UniVLG bridges the gap between existing 2D-centric models and the rich 3D sensory data available in embodied systems.
  arXiv  Detail & Related papers  (2025-03-13T17:56:22Z)
• FunGraph: Functionality Aware 3D Scene Graphs for Language-Prompted Scene Interaction [1.8124328823188356]
  We aim to develop a representation that enables robots to directly interact with their environment.<n>We focus on detecting and storing objects at a finer resolution, focusing on affordance-relevant parts.<n>We leverage currently available 3D resources to generate 2D data and train a detector, which is then used to augment the standard 3D scene graph generation pipeline.
  arXiv  Detail & Related papers  (2025-03-10T23:13:35Z)
• Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulation [30.744137117668643]
  Lift3D is a framework that enhances 2D foundation models with implicit and explicit 3D robotic representations to construct a robust 3D manipulation policy.<n>In experiments, Lift3D consistently outperforms previous state-of-the-art methods across several simulation benchmarks and real-world scenarios.
  arXiv  Detail & Related papers  (2024-11-27T18:59:52Z)
• Memorize What Matters: Emergent Scene Decomposition from Multitraverse [54.487589469432706]
  We introduce 3D Gaussian Mapping, a camera-only offline mapping framework grounded in 3D Gaussian Splatting. 3DGM converts multitraverse RGB videos from the same region into a Gaussian-based environmental map while concurrently performing 2D ephemeral object segmentation. We build the Mapverse benchmark, sourced from the Ithaca365 and nuPlan datasets, to evaluate our method in unsupervised 2D segmentation, 3D reconstruction, and neural rendering.
  arXiv  Detail & Related papers  (2024-05-27T14:11:17Z)
• SUGAR: Pre-training 3D Visual Representations for Robotics [85.55534363501131]
  We introduce a novel 3D pre-training framework for robotics named SUGAR. SUGAR captures semantic, geometric and affordance properties of objects through 3D point clouds. We show that SUGAR's 3D representation outperforms state-of-the-art 2D and 3D representations.
  arXiv  Detail & Related papers  (2024-04-01T21:23:03Z)
• Homography Loss for Monocular 3D Object Detection [54.04870007473932]
  A differentiable loss function, termed as Homography Loss, is proposed to achieve the goal, which exploits both 2D and 3D information. Our method yields the best performance compared with the other state-of-the-arts by a large margin on KITTI 3D datasets.
  arXiv  Detail & Related papers  (2022-04-02T03:48:03Z)
• Indoor Semantic Scene Understanding using Multi-modality Fusion [0.0]
  We present a semantic scene understanding pipeline that fuses 2D and 3D detection branches to generate a semantic map of the environment. Unlike previous works that were evaluated on collected datasets, we test our pipeline on an active photo-realistic robotic environment. Our novelty includes rectification of 3D proposals using projected 2D detections and modality fusion based on object size.
  arXiv  Detail & Related papers  (2021-08-17T13:30:02Z)
• CRAVES: Controlling Robotic Arm with a Vision-based Economic System [96.56564257199474]
  Training a robotic arm to accomplish real-world tasks has been attracting increasing attention in both academia and industry.<n>This work discusses the role of computer vision algorithms in this field.<n>We present an alternative solution, which uses a 3D model to create a large number of synthetic data.
  arXiv  Detail & Related papers  (2018-12-03T13:28:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.