Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading

• URL: http://arxiv.org/abs/2608.05675v2
• Date: Fri, 07 Aug 2026 04:01:19 GMT
• Title: Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading
• Abstract summary: We act on the complementary relation, equivariance: edit a figure's data and the correct answer must change by a computable amount.<n>Two matched edits are provably complete for affine reading errors; no suite of swap edits is complete for label permutations, and cyclic relabeling closes most of that gap.<n>The predicted ordering holds across three models and a hand-labeled population immune to the one circularity in how it is selected.
• Score: 0.8594140167290097
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Label-free reliability for vision-language models rests on invariance: perturb the input and a faithful reader's answer should not change. This has a known blind spot, a systematic misreading survives the perturbation and gets certified wrong, which we show is computable, not just real: an error is invisible to an edit exactly when the two commute, so the errors a suite cannot reach form its joint centralizer, a set that shrinks as edits are added and can be written down rather than guessed at. We act on the complementary relation, equivariance: edit a figure's data and the correct answer must change by a computable amount. Two matched edits are provably complete for affine reading errors; no suite of swap edits is complete for label permutations, and cyclic relabeling closes most of that gap. We instantiate the theory as the Equivariance-Consistency Score, a label-free, training-free detector, and release REND-EQUIV, pairing matched invariance and equivariance sets over identical data. The predicted ordering holds across three models and a hand-labeled population immune to the one circularity in how it is selected; a second invariance-family method confirms the blind spot belongs to the relation, not to any implementation; and cyclic relabeling delivers its predicted gain on a matched real sample. The same characterization explains a reported inversion of this ordering in the classifier metamorphic-testing literature: detectability is a joint property of the relation and the fault class, never of the relation alone.

Related papers

• Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores [2.2400646874013037]
  Under label shift, standard conformal procedures no longer maintain their intended coverage behavior.<n>We propose the emphLabel-Shift-Adjusted Bayesian Score (LSA score), a nonconformity score derived from a posterior predictive identity.<n>LSA score consistently yields shorter intervals than residual-based and source-based Bayesian scores.
  arXiv  Detail & Related papers  (2026-09-11T03:06:31Z)
• Separating Covariate Shift from Mechanism Change with Two Discriminators: CJSD, a Conditional Discrepancy with an Exact Covariate-Concept Decomposition [0.0]
  Conditional Jensen-Shannon Discrepancy (CJSD)<n>On a ten-measure battery over 202 dataset pairs, only the two conditional-information estimators -- CJSD and a kNN plug-in for the same estimand -- fail.
  arXiv  Detail & Related papers  (2026-08-20T10:48:08Z)
• Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability [51.56484100374058]
  We prove the first identifiability theorem for a mechanistic-interpretability primitive.<n>The spectrum is an identifiable, model-intrinsic fingerprint with a stated error bar, not a legible decomposition.
  arXiv  Detail & Related papers  (2026-08-10T19:42:01Z)
• Forgetting Is Not a Fix: Path Dependence in Sequential Engram Editing [35.76482964927589]
  We test the hypothesis that concept-specific memory traces become linear objects that can be extracted once and combined arithmetically.<n>We run that test on the authors' own reference implementation, at their reported best edit strength.<n>For unlearning-as-compliance: erasure certified today does not certify the artifact after its next edit.
  arXiv  Detail & Related papers  (2026-07-06T23:45:06Z)
• Anytime-Valid Confirmation of Label-Shift Corrections [0.6599344783327052]
  In small-batch scientific deployments, labeled target outcomes may be too scarce for reliable shift estimation.<n>We show that the per-observation likelihood ratio between a label-shift-corrected predictive and the source predictive is a conditional e-value.
  arXiv  Detail & Related papers  (2026-06-12T02:03:21Z)
• Decoding in Order-Agnostic Language Models: Chain-Rule Deviation and Uniform Spreading [0.6916773850242582]
  Order-agnostic language models (OALMs) are trained to predict masked tokens under arbitrary conditioning sets.<n>We show that learning conditionals are not exact factorizations of a coherent joint distribution.<n>We propose a complementary diagnostic based on the shape of the confidence trace.
  arXiv  Detail & Related papers  (2026-05-31T04:25:36Z)
• Bootstrapping with AI/ML-generated labels [0.0]
  We show that a seemingly natural fixed-label bootstrap, which generates data using estimated labels but relies on a corrupted version in estimation, is generally invalid unless a strong independence condition holds.<n>We propose a coupled-label bootstrap that jointly resamples the true and imputed labels, and show it is valid without this condition.<n>We illustrate the methods in simulations and apply them to investigate the relationship between wages and remote work status.
  arXiv  Detail & Related papers  (2026-04-26T15:48:03Z)
• Latent Veracity Inference for Identifying Errors in Stepwise Reasoning [78.29317733206643]
  We introduce Veracity Search (VS), a discrete search algorithm over veracity assignments.<n>It performs otherwise intractable inference in the posterior distribution over latent veracity values.<n>It generalizes VS, enabling accurate zero-shot veracity inference in novel contexts.
  arXiv  Detail & Related papers  (2025-05-17T04:16:36Z)
• Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression [102.24287051757469]
  We study self-supervised covariance estimation in deep heteroscedastic regression.<n>We derive an upper bound on the 2-Wasserstein distance between normal distributions.<n>Experiments over a wide range of synthetic and real datasets demonstrate that the proposed 2-Wasserstein bound coupled with pseudo label annotations results in a computationally cheaper yet accurate deep heteroscedastic regression.
  arXiv  Detail & Related papers  (2025-02-14T22:37:11Z)
• Training on Test Data with Bayesian Adaptation for Covariate Shift [96.3250517412545]
  Deep neural networks often make inaccurate predictions with unreliable uncertainty estimates. We derive a Bayesian model that provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters. We show that our method improves both accuracy and uncertainty estimation.
  arXiv  Detail & Related papers  (2021-09-27T01:09:08Z)
• Counterfactual Invariance to Spurious Correlations: Why and How to Pass Stress Tests [87.60900567941428]
  A spurious correlation' is the dependence of a model on some aspect of the input data that an analyst thinks shouldn't matter. In machine learning, these have a know-it-when-you-see-it character. We study stress testing using the tools of causal inference.
  arXiv  Detail & Related papers  (2021-05-31T14:39:38Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)
• Distribution-free binary classification: prediction sets, confidence intervals and calibration [106.50279469344937]
  We study three notions of uncertainty quantification -- calibration, confidence intervals and prediction sets -- for binary classification in the distribution-free setting. We derive confidence intervals for binned probabilities for both fixed-width and uniform-mass binning. As a consequence of our 'tripod' theorems, these confidence intervals for binned probabilities lead to distribution-free calibration.
  arXiv  Detail & Related papers  (2020-06-18T14:17:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.