Answer-Level Trust Selection for Physical Vision-Language Reasoning

• URL: http://arxiv.org/abs/2608.19807v1
• Date: Thu, 20 Aug 2026 09:00:17 GMT
• Title: Answer-Level Trust Selection for Physical Vision-Language Reasoning
• Abstract summary: We propose Answer-Level Trust Selection (ATS) for accepting or rejecting individual VLM predictions.<n>ATS requires no fine-tuning, auxiliary verifier, or access to the model's internal logits.<n>We evaluate ATS in depth on Qwen2.5-VL-7B and across 20 VLM backbones.
• Score: 19.884429996574763
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) can estimate physical quantities such as duration, speed, and acceleration from visual observations, but existing benchmarks primarily assess overall model performance against annotated ground truth. In deployment, a key question is whether an individual prediction can be trusted when its ground truth is unavailable. Self-consistency alone may fail to capture important failure modes: a VLM may produce stable-but-wrong estimates or rely on textual priors rather than visual evidence. We formulate answer-level selective prediction for quantitative physical reasoning and propose Answer-Level Trust Selection (ATS), a post-hoc, model-agnostic framework for accepting or rejecting individual VLM predictions. ATS requires no fine-tuning, auxiliary verifier, or access to the model's internal logits. Instead, it aggregates eight interpretable behavioral diagnostic scores derived from repeated queries and controlled interventions into a unified trust score. We evaluate ATS in depth on Qwen2.5-VL-7B and across 20 VLM backbones, examining selective performance, diagnostic behavior, and targeted failure modes. Our results show that intervention-based diagnostics help identify stable-but-wrong and prior-tracking predictions that repeated agreement alone may miss. However, improved failure-case rejection can come at the cost of lower retention of correct predictions. ATS therefore complements model-level capability evaluation with answer-level reliability assessment for quantitative VLM predictions. Code will be released upon publication.

Related papers

• ENCP: Episode-Normalized Conformal Prediction for Vision-and-Language Navigation [1.6640915052639702]
  Uncertainty estimation for Vision-Language-Navigation (VLN) models is a critical task since it can help identify ambiguous and unreliable predictions.<n>We propose Episode-Normalized Conformal Prediction (ENCP) which rescales a nonconformity score by the policy's residual confidence and calibrates one maximum score per episode.
  arXiv  Detail & Related papers  (2026-09-15T17:42:15Z)
• EVADE: Evidence-Verified Agentic Diagnosis with Escape [0.880899367147235]
  EVADE is an inferential, non-training method that enhances the safety of deploying a single frozen VLM.<n>It commits only when both the entire image and the zoomed view responses agree.<n>It reduces expected calibration error (ECE) by up to 45% compared to zero-shot.
  arXiv  Detail & Related papers  (2026-08-19T11:51:24Z)
• Can Vision Language Models Judge Action Quality? An Empirical Evaluation [5.688409551177917]
  Action Quality Assessment (AQA) has broad applications in physical therapy, sports coaching, and competitive judging.<n>Although Vision Language Models (VLMs) hold considerable promise for AQA, their actual performance in this domain remains largely uncharacterised.<n>We present a comprehensive evaluation of state-of-the-art VLMs across activity domains (e.g. fitness, figure skating, diving), tasks, representations, and prompting strategies.
  arXiv  Detail & Related papers  (2026-04-09T14:29:19Z)
• Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models [94.68358825189738]
  We propose a post-training framework for verifiable domains that jointly optimize prediction accuracy and reasoning quality.<n>We evaluate VPS on chess, a controlled testbed where reasoning steps can be deterministically verified against engine signals.<n>VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation.
  arXiv  Detail & Related papers  (2026-04-03T15:19:46Z)
• VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation [22.921677603408188]
  Large Vision-Language Models (LVLMs) frequently hallucinate, limiting their safe deployment in real-world applications.<n>We propose VAUQ, a vision-aware uncertainty quantification framework for LVLM self-evaluation.<n>VAUQ explicitly measures how strongly a model's output depends on visual evidence.
  arXiv  Detail & Related papers  (2026-02-24T16:11:14Z)
• On Calibration of Large Language Models: From Response To Capability [66.59139960234326]
  Large language models (LLMs) are widely deployed as general-purpose problem solvers.<n>We introduce capability calibration, which targets the model's expected accuracy on a query.<n>Our results demonstrate that capability-calibrated confidence improves pass@$k$ prediction and inference budget allocation.
  arXiv  Detail & Related papers  (2026-02-14T01:07:45Z)
• Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoning [71.19675094463834]
  This work introduces Counterfactual VLA, a self-reflective VLA framework that enables the model to reason about and revise its planned actions before execution.<n>CF-VLA first generates time-segmented meta-actions that summarize driving intent, and then performs counterfactual reasoning conditioned on both the meta-actions and the visual context.<n> Experiments on large-scale driving datasets show that CF-VLA improves trajectory accuracy by up to 17.6%, enhances safety metrics by 20.5%, and exhibits adaptive thinking.
  arXiv  Detail & Related papers  (2025-12-30T19:04:17Z)
• Explanation-Driven Counterfactual Testing for Faithfulness in Vision-Language Model Explanations [0.8657627742603715]
  Vision-Language Models (VLMs) often produce fluent Natural Language Explanations (NLEs) that sound convincing but may not reflect causal factors driving predictions.<n>This mismatch of plausibility and faithfulness poses technical and governance risks.<n>We introduce Explanation-Driven Counterfactual Testing (EDCT), a fully automated verification procedure for a target VLM.
  arXiv  Detail & Related papers  (2025-09-27T15:16:23Z)
• Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction [13.958280616597385]
  This work presents the first framework to analyze the uncertainty by offering a prediction interval of LLM-based scoring via conformal prediction.<n>We perform extensive experiments and analysis, which show that conformal prediction can provide valid prediction interval with coverage guarantees.
  arXiv  Detail & Related papers  (2025-09-23T05:26:28Z)
• PredictaBoard: Benchmarking LLM Score Predictability [50.47497036981544]
  Large Language Models (LLMs) often fail unpredictably.<n>This poses a significant challenge to ensuring their safe deployment.<n>We present PredictaBoard, a novel collaborative benchmarking framework.
  arXiv  Detail & Related papers  (2025-02-20T10:52:38Z)
• A Probabilistic Perspective on Unlearning and Alignment for Large Language Models [48.96686419141881]
  We introduce the first formal probabilistic evaluation framework for Large Language Models (LLMs)<n> Namely, we propose novel metrics with high probability guarantees concerning the output distribution of a model.<n>Our metrics are application-independent and allow practitioners to make more reliable estimates about model capabilities before deployment.
  arXiv  Detail & Related papers  (2024-10-04T15:44:23Z)
• Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning [67.82016092549284]
  We introduce ReCoVERR, an inference-time algorithm to reduce the over-abstention of a selective vision-language system. ReCoVERR tries to find relevant clues in an image that provide additional evidence for the prediction.
  arXiv  Detail & Related papers  (2024-02-23T21:16:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.