Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs

• URL: http://arxiv.org/abs/2605.19528v1
• Date: Tue, 19 May 2026 08:30:21 GMT
• Title: Towards Camera-Robust 3D Localization: Equation-Anchored Tool-Use for MLLMs
• Abstract summary: 3D localization in Multimodal Large Language Models (MLLMs) is limited by camera intrinsic ambiguity.<n>We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables.<n>Our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale.
• Score: 72.8641426724502
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D localization in Multimodal Large Language Models (MLLMs), including 3D object detection and 3D visual grounding, is fundamentally limited by camera intrinsic ambiguity: the same image admits different 3D scenes under different cameras. Existing MLLMs either ignore camera parameters and overfit to a canonical training intrinsic, or retrieve depth and 3D cues from external tools but treat the returned values as reference cues (numerical hints that the model is free to interpret implicitly), both preventing camera information from being deterministically propagated into the prediction. We propose an equation-anchored tool-use framework that re-purposes spatial tools as formula variables. The proposed framework proactively retrieves camera intrinsics and samples multi-point metric depths, writes the pinhole back-projection equation $\hat{X} = (u_c - c_x)\bar{Z}/f_x$ explicitly in Chain-of-Thought (CoT), and substitutes tool outputs into the formula before regressing the final 9-DoF bounding box. On both 3D object detection and 3D visual grounding tasks under rescaled camera intrinsics from $0.5\times$ to $1.5\times$, our method outperforms RGB-only and tool-augmented baselines, with significant gains where the camera deviates most from the training scale. Code and data will be released.

Related papers

• SAM3D-Guided Object-Centric Representation Alignment for Vision-Language-Action Models [3.2119871428711875]
  We propose an object-centric 3D representation alignment framework to provide target-object 3D priors during training.<n>Specifically, we localize task-relevant objects with object recognition models, generate corresponding object masks, and use SAM3D to extract dense object-level 3D representations.<n>Experiments show consistent improvements, achieving 99.1% on LIBERO and an average length of 4.11 on CALVIN.
  arXiv  Detail & Related papers  (2026-07-28T16:05:32Z)
• LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sight [105.9472902251177]
  We present a VLM-native recipe that casts 3D detection as a next-token prediction problem.<n>Our model achieves state-of-the-art results, with 49.89 AP_3D, surpassing the previous best by +15.51 absolute improvement.
  arXiv  Detail & Related papers  (2025-11-25T18:59:45Z)
• LLMI3D: MLLM-based 3D Perception from a Single 2D Image [77.13869413871028]
  multimodal large language models (MLLMs) excel in general capacity but underperform in 3D tasks.<n>In this paper, we propose solutions for weak 3D local spatial object perception, poor text-based geometric numerical output, and inability to handle camera focal variations.<n>We employ parameter-efficient fine-tuning for a pre-trained MLLM and develop LLMI3D, a powerful 3D perception MLLM.
  arXiv  Detail & Related papers  (2024-08-14T10:00:16Z)
• Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos [15.532504015622159]
  Category-level 3D pose estimation is a fundamentally important problem in computer vision and robotics. We tackle the problem of learning to estimate the category-level 3D pose only from casually taken object-centric videos.
  arXiv  Detail & Related papers  (2024-07-05T09:43:05Z)
• Monocular 3D Object Detection with Depth from Motion [74.29588921594853]
  We take advantage of camera ego-motion for accurate object depth estimation and detection. Our framework, named Depth from Motion (DfM), then uses the established geometry to lift 2D image features to the 3D space and detects 3D objects thereon. Our framework outperforms state-of-the-art methods by a large margin on the KITTI benchmark.
  arXiv  Detail & Related papers  (2022-07-26T15:48:46Z)
• MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries [18.70932813595532]
  3D tracking from multiple cameras is a key component in a vision-based autonomous driving system. We propose an end-to-end textbfMUlti-camera textbfTRacking framework called MUTR3D. MUTR3D does not explicitly rely on the spatial and appearance similarity of objects. It outperforms state-of-the-art methods by 5.3 AMOTA on the nuScenes dataset.
  arXiv  Detail & Related papers  (2022-05-02T01:45:41Z)
• DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries [43.02373021724797]
  We introduce a framework for multi-camera 3D object detection. Our method manipulates predictions directly in 3D space. We achieve state-of-the-art performance on the nuScenes autonomous driving benchmark.
  arXiv  Detail & Related papers  (2021-10-13T17:59:35Z)
• MonoCInIS: Camera Independent Monocular 3D Object Detection using Instance Segmentation [55.96577490779591]
  Methods need to have a degree of 'camera independence' in order to benefit from large and heterogeneous training data. We show that more data does not automatically guarantee a better performance, but rather, methods need to have a degree of 'camera independence' in order to benefit from large and heterogeneous training data.
  arXiv  Detail & Related papers  (2021-10-01T14:56:37Z)
• Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation [57.11299763566534]
  We present a solution to recover 3D pose from multi-view images captured with spatially calibrated cameras. We exploit 3D geometry to fuse input images into a unified latent representation of pose, which is disentangled from camera view-points. Our architecture then conditions the learned representation on camera projection operators to produce accurate per-view 2d detections.
  arXiv  Detail & Related papers  (2020-04-05T12:52:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.