Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models

• URL: http://arxiv.org/abs/2603.06828v1
• Date: Fri, 06 Mar 2026 19:43:26 GMT
• Title: Step-Level Visual Grounding Faithfulness Predicts Out-of-Distribution Generalization in Long-Horizon Vision-Language Models
• Abstract summary: We show that models that maintain temporally grounded beliefs generalize better.<n>We formalize this as behavioral faithfulness over long horizons.<n>We show that temporal grounding quality is a leading indicator of robustness.
• Score: 0.7919969809015935
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We uncover a behavioral law of long-horizon vision-language models: models that maintain temporally grounded beliefs generalize better. Standard benchmarks measure only final-answer accuracy, which obscures how models use visual information; a model can guess correctly while its step-by-step reasoning is entirely unanchored to the visual input. We formalize this as behavioral faithfulness over long horizons, an empirically measurable property that quantifies whether a model's intermediate reasoning remains consistent with the evolving visual state. Across eight models on three long-horizon benchmarks, we demonstrate that temporal grounding quality is a leading indicator of robustness: the Step Grounding Rate (SGR) predicts out-of-distribution retention with $r = 0.83$ (permutation test $p = 0.003$), a relationship that holds within capacity-matched models and cannot be explained by scale or in-distribution accuracy. Critically, grounding quality varies by up to 10.8 percentage points within parameter-matched 7B models despite similar accuracy, revealing it as an independent axis of model capability. Multiple robustness checks confirm the signal reflects genuine visual reliance: counterfactual traces drop SGR by 26--41 percentage points, cross-architecture verifiers agree at $ρ= 0.96$, random reasoning scores near chance ($\sim 18\%$), and the predictor remains strong even without explicit reasoning disclosure ($r = 0.78$).

Related papers

• AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow [13.11051449383848]
  We present textbfAffectFlow-DINO, a multi-task learning system for the 11th ABAW challenge.<n>It extends a standard deterministic architecture with a conditional rectified-flow head to model the inherent ambiguity of in-the-wild facial behavior.
  arXiv  Detail & Related papers  (2026-07-14T20:31:21Z)
• Paying More Attention to Visual Tokens in Self-Evolving Large Multimodal Models [54.097381112138]
  Current self-evolving LMMs struggle on vision-language understanding tasks such as image captioning and visual question answering.<n>We propose VISE, a purely unsupervised self-evolving framework that directly regularizes the model's visual conditioning policy.<n>VISE operates within a single model without specialist roles, external reward models, or annotations.
  arXiv  Detail & Related papers  (2026-06-25T17:59:55Z)
• CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction [51.56484100374058]
  Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's behavioral robustness.<n>We introduce textscCaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction.<n>textscCaliDist consistently achieves lower Expected Error (ECE) and Brier Score compared with strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T07:27:53Z)
• Consistent Yet Wrong: Evidence Insensitivity in Spatial Vision-Language Models [29.66841995436342]
  Modern vision-language models (VLMs) remain unreliable on metric distance queries.<n>We introduce textbfViewDiag, a controlled multi-view evaluation protocol built from Hypersim, ScanNet, and KITTI360.<n>We show that stable predictions may reflect prior-driven collapse rather than evidence-sensitive reasoning.
  arXiv  Detail & Related papers  (2026-06-01T18:06:08Z)
• Capability $\ eq$ Interpretability: Human Interpretability of Vision Foundation Models [24.250649573452666]
  We build a framework for measuring and comparing the human interpretability of vision models.<n>A chance-anchored scoring function places every model on a common scale.<n>Foundation models are consistently *less* interpretable than their supervised counterparts.
  arXiv  Detail & Related papers  (2026-05-19T18:00:06Z)
• Epistemic Observability in Language Models [0.0]
  We find that models report highest confidence precisely when they are fabricating.<n>We prove, under formal assumptions, that this is not a capability gap but an observational one.<n>We construct a tensor interface that escapes the impossibility by exporting computational byproducts.
  arXiv  Detail & Related papers  (2026-03-20T21:59:34Z)
• Scaling Laws for Moral Machine Judgment in Large Language Models [0.0]
  We evaluate 75 large language model configurations (0.27B--1000B parameters) using the Moral Machine framework.<n>We observe a consistent power-law relationship with distance from human preferences decreasing as model size.<n> Extended reasoning models show significantly better alignment, with this effect being more pronounced in smaller models.
  arXiv  Detail & Related papers  (2026-01-25T00:12:45Z)
• When Small Models Are Right for Wrong Reasons: Process Verification for Trustworthy Agents [0.0]
  We reveal a critical reliability crisis: 50-69% of correct answers from small language models contain fundamentally flawed reasoning.<n>We introduce the Reasoning Integrity Score (RIS), a process-based metric validated with substantial inter-rater agreement.<n>We show RAG succeeds by grounding calculations in external evidence, reducing errors by 7.6%, while meta-cognition amplifies confusion without sufficient model capacity.
  arXiv  Detail & Related papers  (2026-01-01T23:54:15Z)
• Measuring Language Model Hallucinations Through Distributional Correctness [7.106986689736826]
  A novel evaluation metric, the Distributional Correctness Score (DCS), is introduced to solve this problem.<n>DCS distinguishes between harmful overconfidence in wrong answers and uncertainty expressed through abstention, providing scores in an interpretable default range.<n>DCS is demonstrated to offer a more nuanced and aligned evaluation paradigm that incentivises models to express genuine uncertainty rather than guessing.
  arXiv  Detail & Related papers  (2025-10-05T17:50:42Z)
• Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models [61.467781476005435]
  skill-wise performance is obscured when inspecting aggregate accuracy, under-utilizing the rich signal modern benchmarks contain. We propose an automatic approach to recover the underlying skills relevant for any evaluation instance, by way of inspecting model-generated rationales. Our skill-slices and framework open a new avenue in model evaluation, leveraging skill-specific analyses to unlock a more granular and actionable understanding of model capabilities.
  arXiv  Detail & Related papers  (2024-10-17T17:51:40Z)
• Selective Learning: Towards Robust Calibration with Dynamic Regularization [79.92633587914659]
  Miscalibration in deep learning refers to there is a discrepancy between the predicted confidence and performance. We introduce Dynamic Regularization (DReg) which aims to learn what should be learned during training thereby circumventing the confidence adjusting trade-off.
  arXiv  Detail & Related papers  (2024-02-13T11:25:20Z)
• Self-Consistency of Large Language Models under Ambiguity [4.141513298907867]
  This work presents an evaluation benchmark for self-consistency in cases of under-specification. We conduct a series of behavioral experiments on the OpenAI model suite using an ambiguous integer sequence completion task. We find that average consistency ranges from 67% to 82%, far higher than would be predicted if a model's consistency was random.
  arXiv  Detail & Related papers  (2023-10-20T11:57:56Z)
• VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives [84.48039784446166]
  We show that model FI supervision can meaningfully improve VQA model accuracy as well as performance on several Right-for-the-Right-Reason metrics. Our best performing method, Visual Feature Importance Supervision (VisFIS), outperforms strong baselines on benchmark VQA datasets. Predictions are more accurate when explanations are plausible and faithful, and not when they are plausible but not faithful.
  arXiv  Detail & Related papers  (2022-06-22T17:02:01Z)
• RobustBench: a standardized adversarial robustness benchmark [84.50044645539305]
  Key challenge in benchmarking robustness is that its evaluation is often error-prone leading to robustness overestimation. We evaluate adversarial robustness with AutoAttack, an ensemble of white- and black-box attacks. We analyze the impact of robustness on the performance on distribution shifts, calibration, out-of-distribution detection, fairness, privacy leakage, smoothness, and transferability.
  arXiv  Detail & Related papers  (2020-10-19T17:06:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.