PointRL: Learning Point-Level Vision-Language Grounding from Verifiable Annotation Evidence

• URL: http://arxiv.org/abs/2608.25299v1
• Date: Wed, 26 Aug 2026 02:10:29 GMT
• Title: PointRL: Learning Point-Level Vision-Language Grounding from Verifiable Annotation Evidence
• Abstract summary: This work presents PointRL, a verifiable reinforcement learning framework that learns point-level grounding.<n>PointRL converts bounding boxes, masks, and instance labels into pointing instructions, while retaining their target supports, instance membership, and set constraints.<n>The proposed reward evaluates parseability, point validity, instance coverage, cardinality consistency, and redundant or missing predictions.
• Score: 4.87376023514389
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models (VLMs) increasingly rely on point coordinates as a compact and executable interface for visual grounding in GUI interaction, robotic manipulation, and interactive visual systems. However, learning reliable pointing behavior remains difficult because the supervision space is inherently non-unique: many coordinates may be valid within the same target region, while multi-instance instructions require target coverage, count consistency, and duplicate suppression. This work presents PointRL, a verifiable reinforcement learning framework that learns point-level grounding from existing heterogeneous annotation evidence. PointRL converts bounding boxes, masks, and instance labels into pointing instructions, while retaining their target supports, instance membership, and set constraints as hidden verifier evidence, i.e., annotations kept outside the prompt and used by a deterministic checker to score predictions. The proposed reward evaluates parseability, point validity, instance coverage, cardinality consistency, and redundant or missing predictions. On PointArena, PointRL improves the overall accuracy of Qwen3.5-4B from 56.11% to 65.58%. Further evaluations on RoboSpatial, BLINK, and Ref-Adv show same-backbone gains on the evaluated external benchmarks, suggesting that verifiable point-level feedback may benefit spatial grounding in these settings.

Related papers

• CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding [56.30142869506262]
  Embodied Reference Understanding involves predicting the object that a person in the scene is referring to through both pointing gesture and language.<n>We propose a dual-model framework, where one model learns from the head-to-fingertip direction and the other from the wrist-to-fingertip direction.<n>We validate our approach through extensive experiments and analysis on the benchmark YouRefIt dataset, achieving an improvement of approximately 4 mAP at the 0.25 IoU threshold.
  arXiv  Detail & Related papers  (2025-07-29T15:00:21Z)
• Jamais Vu: Exposing the Generalization Gap in Supervised Semantic Correspondence [37.26437707181298]
  We propose a novel approach for learning dense correspondences by lifting 2D keypoints into a canonical 3D space using monocular depth estimation.<n>Our method constructs a continuous canonical manifold that captures object geometry without requiring explicit 3D supervision or camera annotations.
  arXiv  Detail & Related papers  (2025-06-09T20:40:47Z)
• VAPO: Visibility-Aware Keypoint Localization for Efficient 6DoF Object Pose Estimation [52.81869878956534]
  Localizing 3D keypoints in a 2D image is an effective way to establish 3D-2D correspondences for instance-level 6DoF object pose estimation.<n>In this paper, we address this issue by localizing the important keypoints in terms of visibility.<n>We construct VAPO (Visibility-Aware POse estimator) by integrating the visibility-aware importance with a state-of-the-art pose estimation algorithm.
  arXiv  Detail & Related papers  (2024-03-21T16:59:45Z)
• Point-Teaching: Weakly Semi-Supervised Object Detection with Point Annotations [81.02347863372364]
  We present Point-Teaching, a weakly semi-supervised object detection framework. Specifically, we propose a Hungarian-based point matching method to generate pseudo labels for point annotated images. We propose a simple-yet-effective data augmentation, termed point-guided copy-paste, to reduce the impact of the unmatched points.
  arXiv  Detail & Related papers  (2022-06-01T07:04:38Z)
• Object Localization under Single Coarse Point Supervision [107.46800858130658]
  We propose a POL method using coarse point annotations, relaxing the supervision signals from accurate key points to freely spotted points. CPR constructs point bags, selects semantic-correlated points, and produces semantic center points through multiple instance learning (MIL) In this way, CPR defines a weakly supervised evolution procedure, which ensures training high-performance object localizer under coarse point supervision.
  arXiv  Detail & Related papers  (2022-03-17T14:14:11Z)
• SASA: Semantics-Augmented Set Abstraction for Point-based 3D Object Detection [78.90102636266276]
  We propose a novel set abstraction method named Semantics-Augmented Set Abstraction (SASA) Based on the estimated point-wise foreground scores, we then propose a semantics-guided point sampling algorithm to help retain more important foreground points during down-sampling. In practice, SASA shows to be effective in identifying valuable points related to foreground objects and improving feature learning for point-based 3D detection.
  arXiv  Detail & Related papers  (2022-01-06T08:54:47Z)
• Attend to Who You Are: Supervising Self-Attention for Keypoint Detection and Instance-Aware Association [40.78849763751773]
  This paper presents a new method to solve keypoint detection and instance association by using Transformer. We propose a novel approach of supervising self-attention for multi-person keypoint detection and instance association.
  arXiv  Detail & Related papers  (2021-11-25T03:41:41Z)
• Learning to Detect Instance-level Salient Objects Using Complementary Image Labels [55.049347205603304]
  We present the first weakly-supervised approach to the salient instance detection problem. We propose a novel weakly-supervised network with three branches: a Saliency Detection Branch leveraging class consistency information to locate candidate objects; a Boundary Detection Branch exploiting class discrepancy information to delineate object boundaries; and a Centroid Detection Branch using subitizing information to detect salient instance centroids.
  arXiv  Detail & Related papers  (2021-11-19T10:15:22Z)
• Oriented RepPoints for Aerial Object Detection [10.818838437018682]
  In this paper, we propose a novel approach to aerial object detection, named Oriented RepPoints. Specifically, we suggest to employ a set of adaptive points to capture the geometric and spatial information of the arbitrary-oriented objects. To facilitate the supervised learning, the oriented conversion function is proposed to explicitly map the adaptive point set into an oriented bounding box.
  arXiv  Detail & Related papers  (2021-05-24T06:18:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.