SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs

• URL: http://arxiv.org/abs/2606.20244v2
• Date: Fri, 19 Jun 2026 09:38:08 GMT
• Title: SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs
• Abstract summary: We present SPOT-E, a test-time method that produces question-conditioned spotlights, optimized per instance via light-weight tuning.<n> SPOT-E yields consistent gains and improved robustness under visual corruptions.
• Score: 74.89208610190595
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) often underperform on evidence intensive tasks because decisive visual evidence are small, localized, and easy to overlook, leading to failures in evidence readout even when high-level reasoning is intact. Prior inference-time visual interventions can improve grounding without retraining, but they are largely open-loop and lack a mechanism to verify whether highlighted evidence is actually used. We study answer-span prediction entropy as a model-internal feedback signal and show that naive entropy minimization is ambiguous, since low entropy may arise from evidence-grounded confidence or shortcut collapse. To resolve this ambiguity, we introduce low-entropy anchors and an entropy-shaping objective that reduces answer uncertainty while preserving baseline high-confidence tokens. We instantiate this principle in SPOT-E, a plug-and-play test-time method that produces question-conditioned spotlights, optimized per instance via light-weight tuning based on Group Relative Policy Optimization (GRPO). Across all benchmarks and different VLM families, SPOT-E yields consistent gains and improved robustness under visual corruptions. Code is publicly available at: https://github.com/YinBo0927/SPOT-E

Related papers

• Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigation [41.3823869123186]
  Multimodal Large Language Models (MLLMs) are prone to hallucination as their generation preferences are insufficiently calibrated to visual evidence.<n>We propose Oriented Pickup Preference Optimization (textttOPPO), an evidence-aware alignment objective that learns preferences over the strength of visual evidence.
  arXiv  Detail & Related papers  (2026-06-29T05:33:22Z)
• Gradient-Guided Reward Optimization for Inference-time Alignment [22.28081683932275]
  We introduce a lightweight inference-time method that performs targeted, minimal intervention during decoding via gradient guidance.<n>Experiments show that GGRO consistently improves inference-time alignment across safety, helpfulness, and reasoning benchmarks.<n>It also increases coverage of high-quality responses and robustness to reward hacking, with minimal computational overhead.
  arXiv  Detail & Related papers  (2026-06-08T15:33:13Z)
• Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection [68.2118210672375]
  We introduce VEPO (Vision-Entropy token-selection for Policy Optimization), an effective RL framework explicitly integrating visual sensitivity with token entropy.<n>Our experiments demonstrate VEPO's leading performance, significantly outperforming the entropy-only baseline by 2.28 points at 7B-scale and 3.15 points at 3B-scale.
  arXiv  Detail & Related papers  (2026-06-02T17:26:55Z)
• Taming the Thinker: Conditional Entropy Shaping for Adaptive LLM Reasoning [63.36243250685778]
  Conditional Entropy Shaping (CES) is a framework that dynamically controls token-level response entropy.<n>CES consistently improves average accuracy while reducing response length relative to DAPO.
  arXiv  Detail & Related papers  (2026-05-19T04:41:51Z)
• Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models [53.15468578562038]
  We propose an Uncertainty-aware Exploratory Direct Preference Optimization (UE-DPO) method for Multimodal Large Language Models (MLLMs)<n>We first quantify the uncertainty from the model's failure to ground token predictions in the given image.<n>We then encourage more learning pressure on visually deficient tokens in preferred samples, and alleviate the over-penalization of beneficial knowledge in dispreferred samples.
  arXiv  Detail & Related papers  (2026-05-06T13:08:12Z)
• Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models [54.76242207842981]
  We identify video syphancy, a failure mode in which Vid-LLMs retract correct, visually grounded judgments under negation-based gaslighting.<n>Rather than merely changing their answers, the models often fabricate unsupported temporal or spatial explanations to justify incorrect revisions.<n>We introduce GasVideo-1000, a curated benchmark designed to probe videotemporal sycophancy with clear visual grounding and temporal reasoning requirements.
  arXiv  Detail & Related papers  (2026-04-20T06:35:26Z)
• Don't Blink: Evidence Collapse during Multimodal Reasoning [0.0]
  Reasoning VLMs can become more accurate while progressively losing visual grounding as they think.<n>This creates task-conditional danger zones where low-entropy predictions are confident but ungrounded.<n>Using this structure, a targeted vision veto reduces selective risk by up to 1.9 percentage points at 90% coverage.
  arXiv  Detail & Related papers  (2026-04-05T18:01:00Z)
• Grounding the Score: Explicit Visual Premise Verification for Reliable Vision-Language Process Reward Models [8.630726904040781]
  Explicit Visual Premise Verification (EVPV) is a lightweight verification interface that conditions step scoring on the reliability of the visual premises a step depends on.<n>EVPV improves step-level verification and consistently boosts Best-of-N reranking accuracy over strong baselines.
  arXiv  Detail & Related papers  (2026-03-17T08:40:26Z)
• EDIS: Diagnosing LLM Reasoning via Entropy Dynamics [3.858418431840288]
  We show that the emphtemporal evolution of confidence during generation carries richer information than aggregate statistics alone.<n>We introduce the Entropy Dynamics Instability Score (textbfEDIS), a trajectory-level metric quantifying instability in entropy evolution.
  arXiv  Detail & Related papers  (2026-02-01T15:43:50Z)
• From Observations to States: Latent Time Series Forecasting [65.98504021691666]
  We propose Latent Time Series Forecasting (LatentTSF), a novel paradigm that shifts TSF from observation regression to latent state prediction.<n>Specifically, LatentTSF employs an AutoEncoder to project observations at each time step into a higher-dimensional latent state space.<n>Our proposed latent objectives implicitly maximize mutual information between predicted latent states and ground-truth states and observations.
  arXiv  Detail & Related papers  (2026-01-30T20:39:44Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• Think Just Enough: Sequence-Level Entropy as a Confidence Signal for LLM Reasoning [5.37133760455631]
  We introduce a novel entropy-based framework to drive token efficiency in large language models during reasoning tasks.<n>Our approach uses Shannon entropy from token-level logprobs as a confidence signal to enable early stopping.<n>We show that entropy-based confidence calibration represents an emergent property of advanced post-training optimization.
  arXiv  Detail & Related papers  (2025-10-09T12:33:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.