MolmoPoint: Better Pointing for VLMs with Grounding Tokens

• URL: http://arxiv.org/abs/2603.28069v1
• Date: Mon, 30 Mar 2026 06:15:06 GMT
• Title: MolmoPoint: Better Pointing for VLMs with Grounding Tokens
• Abstract summary: We propose a more intuitive pointing mechanism that directly selects the visual tokens that contain the target concept.<n>Our model generates a special pointing token that cross-attends to the input image or video tokens and selects the appropriate one.<n>Using this method, we set a new state-of-the-art on image pointing (70.7% on PointBench), set a new state-of-the-art among fully open models on GUI pointing (61.1% on ScreenSpotPro), and improve video pointing (59.1% human preference win rate vs. a text coordinate baseline) and tracking.
• Score: 54.498877269283206
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Grounding has become a fundamental capability of vision-language models (VLMs). Most existing VLMs point by generating coordinates as part of their text output, which requires learning a complicated coordinate system and results in a high token count. Instead, we propose a more intuitive pointing mechanism that directly selects the visual tokens that contain the target concept. Our model generates a special pointing token that cross-attends to the input image or video tokens and selects the appropriate one. To make this model more fine-grained, we follow these pointing tokens with an additional special token that selects a fine-grained subpatch within the initially selected region, and then a third token that specifies a location within that subpatch. We further show that performance improves by generating points sequentially in a consistent order, encoding the relative position of the previously selected point, and including a special no-more-points class when selecting visual tokens. Using this method, we set a new state-of-the-art on image pointing (70.7% on PointBench), set a new state-of-the-art among fully open models on GUI pointing (61.1% on ScreenSpotPro), and improve video pointing (59.1% human preference win rate vs. a text coordinate baseline) and tracking (+6.3% gain on Molmo2Track). We additionally show that our method achieves much higher sample efficiency and discuss the qualitative differences that emerge from this design change.

Related papers

• Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers [58.582630781164305]
  Visual geometry transformers are powerful architectures for multi-view 3D reconstruction.<n>Their computational cost grows quadratically with the input sequence length due to the global attention layers inside these models.<n>We address this challenge with a simple yet general strategy: restricting the number of key/value tokens that each query interacts with during global attention.<n>Our approach offers a superior speed-accuracy trade-off compared to existing solutions.
  arXiv  Detail & Related papers  (2026-05-22T17:55:13Z)
• When LLaVA Meets Objects: Token Composition for Vision-Language-Models [31.554057603168214]
  Mask-LLaVA is a framework that leverages different levels of visual features to create a compact yet information-rich visual representation for autoregressive Vision Language Models.<n>While all tokens are used during training, it shows that the resulting model can flexibly drop especially the number of mask-based object-tokens at test time.<n>Our analysis demonstrates that combining multi-level features enables efficient learning with fewer tokens while allowing dynamic token selection at test time for good performance.
  arXiv  Detail & Related papers  (2026-02-04T18:50:46Z)
• SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs [59.415473779171315]
  We propose a novel visual token pruning strategy called textbfSaliency-textbfCoverage textbfOriented token textbfPruning for textbfEfficient MLLMs.
  arXiv  Detail & Related papers  (2025-10-28T09:29:37Z)
• Talking Points: Describing and Localizing Pixels [17.428135548304308]
  We introduce a novel framework for pixel level grounding.<n>The framework consists of two complementary components: a Point Descriptor that generates rich, contextual descriptions of individual keypoints, and a Point Localizer that regresses precise pixel coordinates from these descriptions.
  arXiv  Detail & Related papers  (2025-10-16T11:42:03Z)
• Switch-a-View: View Selection Learned from Unlabeled In-the-wild Videos [71.01549400773197]
  We introduce SWITCH-A-VIEW, a model that learns to automatically select the viewpoint to display at each timepoint when creating a how-to video.<n>We pose a pretext task that pseudo-labels segments in the training videos for their primary viewpoint.<n>We then discover the patterns between the visual and spoken content in a how-to video on the one hand and its view-switch moments on the other hand.
  arXiv  Detail & Related papers  (2024-12-24T12:16:43Z)
• Tokenize Anything via Prompting [65.93061853439512]
  We present a unified, promptable model capable of simultaneously segmenting, recognizing, and captioning anything. We train a generalizable model with massive segmentation masks, eg, SA-1B masks, and semantic priors from a pre-trained CLIP model with 5 billion parameters. We believe this model can be a versatile region-level image tokenizer, capable of encoding general-purpose region context.
  arXiv  Detail & Related papers  (2023-12-14T17:01:02Z)
• How can objects help action recognition? [74.29564964727813]
  We investigate how we can use knowledge of objects to design better video models. First, we propose an object-guided token sampling strategy that enables us to retain a small fraction of the input tokens. Second, we propose an object-aware attention module that enriches our feature representation with object information.
  arXiv  Detail & Related papers  (2023-06-20T17:56:16Z)
• Point-McBert: A Multi-choice Self-supervised Framework for Point Cloud Pre-training [6.037383467521294]
  We propose Point-McBert, a pre-training framework with eased and refined supervision signals. Specifically, we ease the previous single-choice constraint on patches, and provide multi-choice token ids for each patch as supervision. Our method achieves 94.1% accuracy on ModelNet40, 84.28% accuracy on the hardest setting of ScanObjectNN and new state-of-the-art performance on few-shot learning.
  arXiv  Detail & Related papers  (2022-07-27T00:34:33Z)
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling [104.82953953453503]
  We present Point-BERT, a new paradigm for learning Transformers to generalize the concept of BERT to 3D point cloud. Experiments demonstrate that the proposed BERT-style pre-training strategy significantly improves the performance of standard point cloud Transformers.
  arXiv  Detail & Related papers  (2021-11-29T18:59:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.