When Correct Decisions Hide Internal Stress: Decision-State Probing in Multimodal Language Models

• URL: http://arxiv.org/abs/2606.08394v1
• Date: Sun, 07 Jun 2026 01:11:36 GMT
• Title: When Correct Decisions Hide Internal Stress: Decision-State Probing in Multimodal Language Models
• Abstract summary: We use S$3$E to analyze behavior-internal decoupling in multimodal language models.<n>We measure whether semantic-conflict candidates induce excess decision-state displacement.<n>Our results suggest forced-choice correctness alone is not a sufficient certificate of invariant internal decision geometry.
• Score: 18.50038551163168
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal language models are typically evaluated through external behavior: selecting the correct image--text match, rejecting unsupported captions, or answering visual queries correctly. However, correct behavior alone does not show that the model's internal decision state remains stable under controlled semantic stress. We study this gap through S$^3$E (Structured Semantic Stress Evaluation), a framework for analyzing behavior-internal decoupling in multimodal language models. S$^3$E uses a positive-anchored A/B forced-choice setup in which an image-supported caption is contrasted against semantic stress candidates under both original and swapped option orders, while hidden states are extracted at the pre-answer decision state. We focus on strict-correct trials, where the model consistently selects the correct caption across both orders. Rather than treating arbitrary hidden-state variation as evidence of instability, we measure whether semantic-conflict candidates induce excess decision-state displacement relative to meaning-preserving controls. Across Qwen3VL, Gemma3, and InternVL3, semantic stress consistently produces positive selected-layer excess displacement over lexical controls despite correct forced-choice behavior, while comparisons against random negatives are model-dependent. We interpret this as a scoped decision-state stress-sensitivity signal rather than evidence of downstream failure or hallucination. Our results suggest that forced-choice correctness alone is not a sufficient certificate of invariant internal decision geometry.

Related papers

• Visual Credit Audit for Multimodal Spatial Reasoning [70.16915309526443]
  Visual Credit Audit separates two Estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence.<n>Applying labels yields dependence-credited correctness (D-CC)<n>Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited.
  arXiv  Detail & Related papers  (2026-07-29T15:55:31Z)
• Calibrating Semantic Uncertainty from Observable Language-Model Probabilities [0.0]
  Language models produce probabilities over words, but professional decisions require uncertainty over meaningful states.<n>We introduce a semantic map: a prespecified, testable bridge from probabilities over verbal responses to probabilities over declared states.<n>We derive posterior-error bounds and conditions for existence, uniqueness, stability and sequential Bayesian updating.
  arXiv  Detail & Related papers  (2026-07-20T00:29:04Z)
• The Computational Basis of Confidence in Large Language Models [2.8220151940446425]
  We use statistical decision confidence to study confidence in language models.<n>We show that answer logits behave as monotonic readouts of a latent decision variable.<n>These results establish SDC as a unifying framework for studying confidence across biological and artificial intelligence.
  arXiv  Detail & Related papers  (2026-07-14T07:24:32Z)
• Behavioural Signatures of Risk-Sensitive Decision-Making in Large Language Models [121.70781851874035]
  Large language models (LLMs) are increasingly used in decision support.<n>It is important to understand whether their choices under uncertainty exhibit stable and interpretable behavioural regularities.<n>Here we examine this question using a controlled multi-model framework based on no-limit Texas Hold'em.
  arXiv  Detail & Related papers  (2026-07-11T10:33:39Z)
• SAGE: Answer-Conditioned Uncertainty Targets for Verbal Uncertainty Alignment [20.5795766462733]
  We propose a group-level uncertainty target that constructs an answer-conditioned uncertainty geometry over sampled responses.<n>Experiments show improved uncertainty ranking, lower calibration error, and reduced overconfidence.
  arXiv  Detail & Related papers  (2026-06-09T23:17:16Z)
• A CAP-like Trilemma for Large Language Models: Correctness, Non-bias, and Utility under Semantic Underdetermination [0.0]
  This paper formulates a CAP-like conjecture for Large Language Models (LLMs)<n>The proposed trilemma states that, under semantic underdetermination, an LLM cannot always simultaneously guarantee strong correctness, strict non-bias, and high utility.
  arXiv  Detail & Related papers  (2026-05-12T07:28:38Z)
• Prompt Sensitivity in Vision-Language Grounding: How Small Changes in Wording Affect Object Detection [6.055168011374501]
  Vision-language models enable open-vocabulary object grounding through natural language queries.<n>We find that overlapping prompts such as "a person," "a human," and "a pedestrian" frequently select different instances.<n> PCA analysis shows this variability is structured and directional, not random.
  arXiv  Detail & Related papers  (2026-04-18T19:55:05Z)
• Adaptive Conformal Prediction for Improving Factuality of Generations by Large Language Models [86.8650252164764]
  Large language models (LLMs) are prone to generating factually incorrect outputs.<n>We propose an adaptive conformal prediction approach that extends conformal score transformation methods to LLMs.<n>This enables prompt-dependent calibration, retaining marginal coverage guarantees while improving conditional coverage.
  arXiv  Detail & Related papers  (2026-04-15T15:35:42Z)
• The Confidence Gate Theorem: When Should Ranked Decision Systems Abstain? [0.0]
  Ranked decision systems must decide when to intervene in ranked outputs and when to abstain.<n>We study when confidence-based abstention monotonically improves decision quality, and when it fails.
  arXiv  Detail & Related papers  (2026-03-10T17:44:10Z)
• Same Answer, Different Representations: Hidden instability in VLMs [65.36933543377346]
  We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness.<n>We apply this framework to modern Vision Language Models (VLMs) across the SEEDBench, MMMU, and POPE datasets.
  arXiv  Detail & Related papers  (2026-02-06T12:24:26Z)
• Explicit Uncertainty Modeling for Active CLIP Adaptation with Dual Prompt Tuning [51.99383151474742]
  We propose a robust uncertainty modeling framework for active CLIP adaptation based on dual-prompt tuning.<n>We show that our method consistently outperforms existing active learning methods under the same annotation budget.
  arXiv  Detail & Related papers  (2026-02-04T09:01:55Z)
• Triangulation as an Acceptance Rule for Multilingual Mechanistic Interpretability [0.0]
  We argue that mechanistic explanations for such models should satisfy a emphcausal standard.<n>Claims must survive causal interventions and must emphcross-reference across environments that perturb surface form while preserving meaning.<n>We ground triangulation in causal abstraction by casting it as an approximate transformation score over a distribution of interchange interventions, connect it to the pragmatic interpretability agenda, and present a comparative experimental protocol across model families, language pairs, and tasks.
  arXiv  Detail & Related papers  (2025-12-31T13:03:34Z)
• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• Conformal Linguistic Calibration: Trading-off between Factuality and Specificity [41.45862052156885]
  We present a framework connecting abstention and linguistic calibration through the lens of linguistic pragmatics.<n>Results demonstrate our method produces calibrated outputs with conformal guarantees on factual accuracy.
  arXiv  Detail & Related papers  (2025-02-26T13:01:49Z)
• Conformal Language Modeling [61.94417935386489]
  We propose a novel approach to conformal prediction for generative language models (LMs) Standard conformal prediction produces prediction sets with rigorous, statistical guarantees. We demonstrate the promise of our approach on multiple tasks in open-domain question answering, text summarization, and radiology report generation.
  arXiv  Detail & Related papers  (2023-06-16T21:55:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.