Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention

• URL: http://arxiv.org/abs/2607.05978v1
• Date: Tue, 07 Jul 2026 08:10:57 GMT
• Title: Propose and Attend: Training-free MLLM Grounding Confidence via Multi-Token Localized Attention
• Abstract summary: Multi-Token Localized Attention (MTLA) is a training-free, post-hoc score that measures how strongly a prediction's tokens attend to the region they claim.<n>Across multiple MLLM families and three modalities, MTLA improves hallucination AUROC by +7 to +38 over the best prior training-free baseline.
• Score: 9.899649326522761
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multimodal large language models can emit localized predictions, bounding boxes for objects and temporal windows for video and audio events, but they hallucinate these regions prolifically. The model's own token log-probabilities are nearly uninformative: they conflate grounding quality with input ambiguity, and coordinate tokens become near-deterministic once the model commits. We propose Multi-Token Localized Attention (MTLA): a training-free, post-hoc score that measures how strongly a prediction's tokens attend to the region they claim. Prior attention-based detectors, which sum attention over the entire input modality and read a single response token, are weaker special cases; we show that summing only within the claimed region and aggregating across all prediction tokens recovers a stronger grounding signal. The same recipe applies almost trivially to other modalities and tasks: object detection in images and temporal localization in video and audio. Across multiple MLLM families and three modalities, MTLA improves hallucination AUROC by +7 to +38 over the best prior training-free baseline. Used as a confidence score for re-ranking, it nearly doubles the zero-shot COCO detection AP of an open-source 8B generalist (from 20.4 to 37.0), narrowing the gap to supervised detectors without any task-specific training.

Related papers

• ActMap: Single-Pass Uncertainty Quantification from Generation-Time Activation Maps [1.7403133838762448]
  ActMap is a representation that compresses the generation-time hidden-state trajectory into a fixed $12times32times128$ of temporal-statistic channels.<n>A lightweight classifier, instantiated as a compact Vision Transformer, reads an estimated correctness probability from each map in a fraction of a millisecond.
  arXiv  Detail & Related papers  (2026-09-10T13:04:24Z)
• Language Models Can Control Their Own Attention [42.07765767561011]
  We introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought.<n>DA significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale.
  arXiv  Detail & Related papers  (2026-09-02T15:43:38Z)
• Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games [69.57330692969543]
  We introduce RNG-Bench, a benchmark suite designed to isolate a base model's ability to reconstruct past observations.<n>RNG-Bench includes two complementary games: Matching Pairs, where card identities briefly revealed at specific locations must later be recalled, and 3D Maze, where egocentric views must be integrated into a spatial map.<n>The hardest configurations require contexts of roughly 128K tokens and 350 image inputs per episode, and remain far from saturated by frontier MLLMs.
  arXiv  Detail & Related papers  (2026-06-17T17:59:34Z)
• Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification [8.07177858013243]
  Self-supervised learning in audio defaults to fine-tuning.<n>We introduce binarized probes: a lightweight and simple pooling method that learns prototypes to perform class-wise information aggregation.<n>Our work establishes probing as a competitive and efficient paradigm for evaluating audio SSL models, challenging the reliance on costly fine-tuning.
  arXiv  Detail & Related papers  (2025-09-29T15:11:18Z)
• Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMs [62.9348974370985]
  We propose attention reallocation (AttnReal) to mitigate hallucinations with nearly zero extra cost.<n>Our approach is motivated by the key observations that, MLLM's unreasonable attention distribution causes features to be dominated by historical output tokens.<n>Based on the observations, AttnReal recycles excessive attention from output tokens and reallocates it to visual tokens, which reduces MLLM's reliance on language priors.
  arXiv  Detail & Related papers  (2025-03-11T11:52:37Z)
• Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection [49.15148871877941]
  Next-token distribution outputs offer a theoretically appealing approach for detection of large language models (LLMs)<n>We propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length.<n>PAWN shows competitive and even better performance in-distribution than the strongest baselines with a fraction of their trainable parameters.
  arXiv  Detail & Related papers  (2025-01-07T17:00:49Z)
• FIRP: Faster LLM inference via future intermediate representation prediction [54.897493351694195]
  FIRP generates multiple tokens instead of one at each decoding step. We conduct extensive experiments, showing a speedup ratio of 1.9x-3x in several models and datasets.
  arXiv  Detail & Related papers  (2024-10-27T15:53:49Z)
• ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models [73.34709921061928]
  We propose a training-free method to inject visual prompts into Multimodal Large Language Models (MLLMs)<n>We optimize a learnable latent variable based on an energy function, enhancing the strength of referring regions in the attention map.<n>Our method offers a promising direction for integrating referring abilities into MLLMs, and supports referring with box, mask, scribble and point.
  arXiv  Detail & Related papers  (2024-07-31T11:40:29Z)
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation [124.9008419182485]
  We present OPERA, a novel MLLM decoding method grounded in an Over-trust Penalty and a Retrospection-Allocation strategy. Our approach begins with an interesting observation that, most hallucinations are closely tied to the knowledge aggregation patterns in the self-attention matrix. Based on the observation, OPERA introduces a penalty term on the model logits during the beam-search decoding to mitigate the over-trust issue.
  arXiv  Detail & Related papers  (2023-11-29T18:57:07Z)
• Semi-Supervised Speech Recognition via Local Prior Matching [42.311823406287864]
  Local prior matching is a semi-supervised objective that distills knowledge from a strong prior. We demonstrate that LPM is theoretically well-native, simple to implement, and superior to existing knowledge distillation techniques.
  arXiv  Detail & Related papers  (2020-02-24T16:07:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.