Revisiting Greedy Decoding for Visual Question Answering: A Calibration Perspective

• URL: http://arxiv.org/abs/2604.23443v1
• Date: Sat, 25 Apr 2026 21:01:05 GMT
• Title: Revisiting Greedy Decoding for Visual Question Answering: A Calibration Perspective
• Abstract summary: Greedy sampling strategies are widely adopted in large language models to balance output coherence and diversity.<n>We show that greedy decoding can be an efficient yet strong default for Visual Questioning (VQA)<n>We propose Reasoning Models, which outperforms both sampling and standard decoding in multimodal reasoning scenarios.
• Score: 16.787456677570127
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Stochastic sampling strategies are widely adopted in large language models (LLMs) to balance output coherence and diversity. These heuristics are often inherited in Multimodal LLMs (MLLMs) without task-specific justification. However, we contend that stochastic decoding can be suboptimal for Visual Question Answering (VQA). VQA is a closed-ended task with head-heavy answer distributions where uncertainty is usually epistemic, arising from missing or ambiguous visual evidence rather than plausible continuations. In this work, we provide a theoretical formalization of the relationship between model calibration and predictive accuracy, and derive the sufficient conditions for greedy decoding optimality. Extensive experiments provide empirical evidence for the superiority of greedy decoding over stochastic sampling across multiple benchmarks. Furthermore, we propose Greedy Decoding for Reasoning Models, which outperforms both stochastic sampling and standard greedy decoding in multimodal reasoning scenarios. Overall, our results caution against naively inheriting LLMs decoding heuristics in MLLMs and demonstrate that greedy decoding can be an efficient yet strong default for VQA.

Related papers

• Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agents [23.69429098915734]
  Verifier-Guided Action Selection (VegAS) is a test-time framework designed to improve robustness of MLLM-based embodied agents.<n>We show that VeGAS consistently improves generalization, achieving up to a 36% relative performance gain over strong CoT baselines.
  arXiv  Detail & Related papers  (2026-05-12T18:08:24Z)
• Task-Aware Calibration: Provably Optimal Decoding in LLMs [46.26328678647223]
  LLM decoding often relies on the model's predictive distribution to generate an output.<n>We introduce task calibration as a paradigm to calibrate the model's predictive distribution in the task-induced latent space.<n>Our work demonstrates that task calibration enables more reliable model decisions across various tasks and applications.
  arXiv  Detail & Related papers  (2026-05-11T08:48:04Z)
• Adaptive Conformal Prediction for Improving Factuality of Generations by Large Language Models [86.8650252164764]
  Large language models (LLMs) are prone to generating factually incorrect outputs.<n>We propose an adaptive conformal prediction approach that extends conformal score transformation methods to LLMs.<n>This enables prompt-dependent calibration, retaining marginal coverage guarantees while improving conditional coverage.
  arXiv  Detail & Related papers  (2026-04-15T15:35:42Z)
• Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Models [52.61023005303122]
  We show that low-confidence remasking improves a proxy for quality while constraining the entropy of the induced sequence distribution.<n>We develop a simple Independent--Hastings sampler that approximately targets this distribution during decoding.
  arXiv  Detail & Related papers  (2026-04-01T02:01:30Z)
• Reaching Beyond the Mode: RL for Distributional Reasoning in Language Models [78.68818219506313]
  This paper describes a multi-answer reinforcement learning approach for training LMs to perform distributional reasoning over multiple answers during inference.<n>Across question-answering, medical diagnostic, and coding benchmarks, we observe improved diversity, coverage, and set-level calibration scores compared to single answer trained baselines.
  arXiv  Detail & Related papers  (2026-03-25T22:20:25Z)
• Quantifying Laziness, Decoding Suboptimality, and Context Degradation in Large Language Models [0.4511923587827302]
  Large Language Models (LLMs) often exhibit behavioral artifacts such as laziness, decoded suboptimality, and context degradation.<n>Our results indicate widespread laziness in satisfying complex multi-part instructions.<n>We suggest strategies to reduce laziness and bolster multi-instruction compliance.
  arXiv  Detail & Related papers  (2025-12-19T03:01:59Z)
• Efficient semantic uncertainty quantification in language models via diversity-steered sampling [46.23327887393273]
  We introduce a diversity-steered sampler that discourages semantically redundant outputs during decoding.<n>Key idea is to inject a continuous semantic-similarity penalty into the model's proposal distribution.<n>Being modular and requiring no gradient access to the base LLM, the framework promises to serve as a drop-in enhancement for uncertainty estimation.
  arXiv  Detail & Related papers  (2025-10-24T10:06:21Z)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs [72.82403830490084]
  We argue that the decoding rule should be calibrated by correctness, not confidence alone.<n>We propose simple strategies that achieve this goal: Greedy-Threshold makes sampling greedy at very low confidence steps.<n>Together, our findings challenge prevailings about decoding under uncertainty and show gains across math and general reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-07T14:46:12Z)
• Semantic uncertainty in advanced decoding methods for LLM generation [35.31962554915952]
  This study investigates semantic uncertainty in large language model (LLM) outputs across different decoding methods.<n>We analyze how different decoding strategies affect both the diversity and reliability of model outputs.
  arXiv  Detail & Related papers  (2025-06-17T10:09:29Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• GRACE: Discriminator-Guided Chain-of-Thought Reasoning [75.35436025709049]
  We propose Guiding chain-of-thought ReAsoning with a CorrectnEss Discriminator (GRACE) to steer the decoding process towards producing correct reasoning steps. GRACE employs a discriminator trained with a contrastive loss over correct and incorrect steps, which is used during decoding to score next-step candidates.
  arXiv  Detail & Related papers  (2023-05-24T09:16:51Z)
• Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs [78.31625291513589]
  We argue that self-consistency is an important criteria for valid multi-step reasoning in tasks where the solution is composed of the answers to multiple sub-steps. We propose two types of self-consistency that are particularly important for multi-step reasoning -- hypothetical consistency and compositional consistency. We demonstrate that multiple variants of the GPT-3/-4 models exhibit poor consistency rates across both types of consistency on a variety of tasks.
  arXiv  Detail & Related papers  (2023-05-23T17:25:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.