Which Reconstruction Model Should a Robot Use? Routing Image-to-3D Models for Cost-Aware Robotic Manipulation

• URL: http://arxiv.org/abs/2603.27797v1
• Date: Sun, 29 Mar 2026 18:23:28 GMT
• Title: Which Reconstruction Model Should a Robot Use? Routing Image-to-3D Models for Cost-Aware Robotic Manipulation
• Abstract summary: Multiple reconstruction methods offer different cost-quality tradeoffs.<n>We propose SCOUT, a novel routing framework that decouples reconstruction scores into two components.<n>SCOUT supports arbitrary cost constraints at inference time, accommodating the multi-dimensional cost constraints common in robotics.
• Score: 14.047294299357352
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robotic manipulation tasks require 3D mesh reconstructions of varying quality: dexterous manipulation demands fine-grained surface detail, while collision-free planning tolerates coarser representations. Multiple reconstruction methods offer different cost-quality tradeoffs, from Image-to-3D models - whose output quality depends heavily on the input viewpoint - to view-invariant methods such as structured light scanning. Querying all models is computationally prohibitive, motivating per-input model selection. We propose SCOUT, a novel routing framework that decouples reconstruction scores into two components: (1) the relative performance of viewpoint-dependent models, captured by a learned probability distribution, and (2) the overall image difficulty, captured by a scalar partition function estimate. As the learned network operates only over the viewpoint-dependent models, view-invariant pipelines can be added, removed, or reconfigured without retraining. SCOUT also supports arbitrary cost constraints at inference time, accommodating the multi-dimensional cost constraints common in robotics. We evaluate on the Google Scanned Objects, BigBIRD, and YCB datasets under multiple mesh quality metrics, demonstrating consistent improvements over routing baselines adapted from the LLM literature across various cost constraints. We further validate the framework through robotic grasping and dexterous manipulation experiments. We release the code and additional results on our website.

Related papers

• KineBench: Benchmarking Embodied World Models via IDM-Free Kinematic Grounding [58.541098216700796]
  evaluating the physical consistency of embodied world models is a critical open challenge.<n>Existing frameworks almost exclusively rely on Inverse Dynamics Models(IDMs) for action extraction.<n>We present KineBench, an IDM-free closed-loop benchmark for EWMs.
  arXiv  Detail & Related papers  (2026-07-22T08:04:17Z)
• Instruct-Particulate: Scaling Feed-Forward 3D Object Articulation with Kinematic Control [86.64250947574749]
  Reconstructing articulated 3D objects is important for animation, gaming, and robotic simulations.<n>Recent neural networks can estimate the articulated structure of 3D objects, but their generalization remains limited by the scarcity of annotated data.<n>We introduce Instruct-Particulate, a model that takes a 3D mesh together with a target kinematic specification.
  arXiv  Detail & Related papers  (2026-06-12T17:59:36Z)
• $μ_0$: A Scalable 3D Interaction-Trace World Model [54.38434730157622]
  We present a scalable world model based on 3D traces.<n>Our TraceExtract system automatically extracts 3D supervision.<n>These results establish 3D traces as a scalable and transferable representation for cross-embodiment manipulation.
  arXiv  Detail & Related papers  (2026-06-11T17:59:56Z)
• Generative Texture Diversification of 3D Pedestrians for Robust Autonomous Driving Perception [0.0]
  In this paper, we propose a method for scaling variability in 3D pedestrian assets for synthetic scene generation.<n>We generate multiple distinct pedestrian instances by synthesizing diverse facial textures and identity-level appearance variations.<n>Using the assets, we construct synthetic datasets and study the impact of mixing real and synthetic data for RGB-based object detection.
  arXiv  Detail & Related papers  (2026-05-13T16:35:50Z)
• Geometrically Consistent Multi-View Scene Generation from Freehand Sketches [58.98194920417429]
  Freehand sketches are the most geometrically impoverished input one could offer a multi-view generator.<n>We address three compounding challenges; absent training data, the need for geometric reasoning from distorted 2D input, and cross-view consistency.<n>Our framework synthesizes all views in a single denoising process without requiring reference images, iterative refinement, or per-scene optimisation.
  arXiv  Detail & Related papers  (2026-04-15T18:00:45Z)
• Differentiable Inverse Graphics for Zero-shot Scene Reconstruction and Robot Grasping [0.820984376071696]
  We introduce a differentiable neuro-graphics model that combines neural foundation models with physics-based differentiable rendering to perform zero-shot scene reconstruction and robot grasping.<n>Our approach offers a pathway towards more data efficient, interpretable and generalizable robot autonomy in novel environments.
  arXiv  Detail & Related papers  (2026-02-04T20:33:50Z)
• D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation [66.7166217399105]
  Embodied agents face a critical dilemma that end-to-end models lack interpretability and explicit 3D reasoning.<n>Our model introduces two key innovations: 1) A Dynamic 3D Chain-of-Thought (3D CoT) that unifies planning, grounding, navigation, and question answering within a single 3D-VLM and CoT pipeline; 2) A Synergistic Learning from Fragmented Supervision (SLFS) strategy, which uses a masked autoregressive loss to learn from massive and partially-annotated hybrid data.
  arXiv  Detail & Related papers  (2025-12-14T09:53:15Z)
• FreeSplatter: Pose-free Gaussian Splatting for Sparse-view 3D Reconstruction [69.63414788486578]
  FreeSplatter is a scalable feed-forward framework that generates high-quality 3D Gaussians from uncalibrated sparse-view images.<n>Our approach employs a streamlined transformer architecture where self-attention blocks facilitate information exchange.<n>We develop two specialized variants--for object-centric and scene-level reconstruction--trained on comprehensive datasets.
  arXiv  Detail & Related papers  (2024-12-12T18:52:53Z)
• Img2CAD: Reverse Engineering 3D CAD Models from Images through VLM-Assisted Conditional Factorization [29.177153478213366]
  Reverse engineering 3D computer-aided design (CAD) models from images is an important task for many downstream applications.<n>In this work, we introduce a novel approach that conditionally factorizes the task into two sub-problems.<n>We propose TrAssembler that, conditioned on the discrete structure with semantics, predicts the continuous attribute values.
  arXiv  Detail & Related papers  (2024-07-19T06:53:30Z)
• SM$^3$: Self-Supervised Multi-task Modeling with Multi-view 2D Images for Articulated Objects [24.737865259695006]
  We propose a self-supervised interaction perception method, referred to as SM$3$, to model articulated objects. By constructing 3D geometries and textures from the captured 2D images, SM$3$ achieves integrated optimization of movable part and joint parameters. Evaluations demonstrate that SM$3$ surpasses existing benchmarks across various categories and objects, while its adaptability in real-world scenarios has been thoroughly validated.
  arXiv  Detail & Related papers  (2024-01-17T11:15:09Z)
• Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data [76.38261311948649]
  Viewset Diffusion is a diffusion-based generator that outputs 3D objects while only using multi-view 2D data for supervision. We train a diffusion model to generate viewsets, but design the neural network generator to reconstruct internally corresponding 3D models. The model performs reconstruction efficiently, in a feed-forward manner, and is trained using only rendering losses using as few as three views per viewset.
  arXiv  Detail & Related papers  (2023-06-13T16:18:51Z)
• Shape, Pose, and Appearance from a Single Image via Bootstrapped Radiance Field Inversion [54.151979979158085]
  We introduce a principled end-to-end reconstruction framework for natural images, where accurate ground-truth poses are not available. We leverage an unconditional 3D-aware generator, to which we apply a hybrid inversion scheme where a model produces a first guess of the solution. Our framework can de-render an image in as few as 10 steps, enabling its use in practical scenarios.
  arXiv  Detail & Related papers  (2022-11-21T17:42:42Z)
• Amodal 3D Reconstruction for Robotic Manipulation via Stability and Connectivity [3.359622001455893]
  Learning-based 3D object reconstruction enables single- or few-shot estimation of 3D object models. Existing 3D reconstruction techniques optimize for visual reconstruction fidelity, typically measured by chamfer distance or voxel IOU. We propose ARM, an amodal 3D reconstruction system that introduces a stability prior over object shapes, (2) a connectivity prior, and (3) a multi-channel input representation.
  arXiv  Detail & Related papers  (2020-09-28T08:52:54Z)
• PaMIR: Parametric Model-Conditioned Implicit Representation for Image-based Human Reconstruction [67.08350202974434]
  We propose Parametric Model-Conditioned Implicit Representation (PaMIR), which combines the parametric body model with the free-form deep implicit function. We show that our method achieves state-of-the-art performance for image-based 3D human reconstruction in the cases of challenging poses and clothing types.
  arXiv  Detail & Related papers  (2020-07-08T02:26:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.