PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought

• URL: http://arxiv.org/abs/2606.24539v1
• Date: Tue, 23 Jun 2026 13:06:51 GMT
• Title: PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought
• Abstract summary: Pointing-based visual grounding requires models to precisely locate target objects by deciphering complex spatial relationships between the visual scene and pointing gestures.<n>We propose PointVG-R, a reasoning-guided Multi-modal Large Language Model (MLLM)<n>We design a novel geometric reasoning pipeline that simulates the iterative cognitive process humans employ when interpreting pointing gestures.
• Score: 19.680475189691965
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pointing-based visual grounding requires models to precisely locate target objects by deciphering complex spatial relationships between the visual scene and pointing gestures. Traditional methods typically encode input images into static feature representations and perform reasoning primarily within the linguistic domain, often overlooking the rich perceptual cues and explicit spatial geometry inherent in images. In this study, we aim to mitigate the cognitive vulnerability of models in interpreting gestural spatial relations by proposing PointVG-R, a reasoning-guided Multi-modal Large Language Model (MLLM). PointVG-R introduces geometric-aware reasoning for pointing-based grounding, enabling the model to think with images through the strategic integration of Reinforcement Learning (RL) and cold-start data. Specifically, we design a novel geometric reasoning pipeline that simulates the iterative cognitive process humans employ when interpreting pointing gestures. Furthermore, we construct EgoPoint-CoT, a high-quality visual Chain-of-Thought (CoT) dataset featuring detailed reasoning trajectories to guide the model via Supervised Fine-Tuning (SFT) and RL. To address the varying quality of learning signals encountered during training, we further propose an Adaptive Importance Weighting strategy based on Group Variance, which dynamically adjusts reward signals to optimize the learning process. Experimental results demonstrate that PointVG-R achieves SOTA performance, outperforming the baseline by $\textbf{15.86}$ points in mIoU. Extensive ablation studies further validate the efficacy of our proposed modules. Code: https://github.com/lingli1724/PointVG-R.

Related papers

• ExACT: Exemplar-Driven Calibrated Refinement for Training-Free Visual Grounding in Remote Sensing Images [51.53960096934913]
  Remote sensing visual grounding (RSVG) aims to locate specific objects in high-resolution RS imagery using free-form natural language descriptions.<n>Recent advances in multimodal large language models (MLLMs) show great potential for such open-vocabulary RSVG.<n>We propose Exemplar-driven Calibrated Refinement (ExACT), a training-free framework driven by a one-shot visual prompting mechanism.
  arXiv  Detail & Related papers  (2026-06-27T13:50:39Z)
• VistaRef: Boosting Visual Spatial Orientation Awareness for Pointing-to-Object Detection [19.163908796279802]
  Grounding deictic gestures in natural images is fundamental to AR and human-robot collaboration.<n> VistaRef is a framework designed to explicitly enhance spatial orientation awareness.
  arXiv  Detail & Related papers  (2026-06-23T12:30:04Z)
• Learning Entropy and Spatial Adaptation Dynamics of Multilayer Perceptrons for Structural Point Extraction [1.4681690787310104]
  This paper extends the concept of Learning Entropy (LE) from temporal adaptive systems to spatial learning in multilayer spatial networks (MLPs)<n>An is trained to predict the intensity of a center pixel from its surrounding spatial context, while LE is evaluated from the incremental adaptation of neural weights during learning across image-derived samples.<n>The Spatial Learning Entropy Maps (SLEM) identify unusual image points and regions that induce strong adaptation of the neural network and therefore have an important role in the learning process.
  arXiv  Detail & Related papers  (2026-06-08T21:05:03Z)
• Implicit Neural Representations: A Signal Processing Perspective [54.57279006229212]
  Implicit neural representations (INRs) mark a fundamental shift in signal modeling, moving from discrete sampled data to continuous functional representations.<n>This article examines the evolution of INRs from a signal processing perspective, emphasizing spectral behavior, sampling theory, and multiscale representation.<n>We highlight the utility of INRs across a broad range of applications, including inverse problems in medical and radar imaging, compression, and 3D scene representation.
  arXiv  Detail & Related papers  (2026-04-16T14:12:06Z)
• Bridging Visual Representation and Reinforcement Learning from Verifiable Rewards in Large Vision-Language Models [35.97638821723309]
  KAWHI is a plug-and-play reward reweighting mechanism that explicitly incorporates structured visual information into uniform reward policy optimization methods.<n>It adaptively localizes semantically salient regions through hierarchical geometric aggregation, identifies vision-critical attention heads via structured attribution, and performs paragraph-level credit reallocation to align spatial visual evidence with semantically decisive reasoning steps.
  arXiv  Detail & Related papers  (2026-03-28T18:40:14Z)
• Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds [53.82500407523346]
  PointINS is an instance-oriented self-supervised framework that enriches point cloud representations through geometry-aware learning.<n>PointINS achieves on average +3.5% mAP improvement for indoor instance segmentation and +4.1% PQ gain for outdoor panoptic segmentation.
  arXiv  Detail & Related papers  (2026-03-26T08:31:06Z)
• RSGround-R1: Rethinking Remote Sensing Visual Grounding through Spatial Reasoning [61.84363374647606]
  Remote Sensing Visual Grounding (RSVG) aims to localize target objects in large-scale aerial imagery based on natural language descriptions.<n>These descriptions often rely heavily on positional cues, posing unique challenges for Multimodal Large Language Models (MLLMs) in spatial reasoning.<n>We propose a reasoning-guided, position-aware post-training framework, dubbed textbfRSGround-R1, to progressively enhance spatial understanding.
  arXiv  Detail & Related papers  (2026-01-29T12:35:57Z)
• Improving Vision-and-Language Reasoning via Spatial Relations Modeling [30.477235227733928]
  Visual commonsense reasoning (VCR) is a challenging multi-modal task. The proposed method can guide the representations to maintain more spatial context. We achieve the state-of-the-art results on VCR and two other vision-and-language reasoning tasks VQA, and NLVR.
  arXiv  Detail & Related papers  (2023-11-09T11:54:55Z)
• INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL [90.06845886194235]
  We propose a modified objective for model-based reinforcement learning (RL) We integrate a term inspired by variational empowerment into a state-space model based on mutual information. We evaluate the approach on a suite of vision-based robot control tasks with natural video backgrounds.
  arXiv  Detail & Related papers  (2022-04-18T23:09:23Z)
• Self-supervised Geometric Perception [96.89966337518854]
  Self-supervised geometric perception is a framework to learn a feature descriptor for correspondence matching without any ground-truth geometric model labels. We show that SGP achieves state-of-the-art performance that is on-par or superior to the supervised oracles trained using ground-truth labels.
  arXiv  Detail & Related papers  (2021-03-04T15:34:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.