Inside-Out: Measuring Generalization in Vision Transformers Through Inner Workings

• URL: http://arxiv.org/abs/2604.08192v1
• Date: Thu, 09 Apr 2026 12:44:19 GMT
• Title: Inside-Out: Measuring Generalization in Vision Transformers Through Inner Workings
• Abstract summary: We focus on two practical scenarios: (1) Before deployment, how to select the best model for unlabeled target data; (2) After deployment, how to monitor model performance under distribution shift.<n>We use the inner workings of a model, i.e., circuits, as a predictive metric of generalization performance.
• Score: 21.672670192853506
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reliable generalization metrics are fundamental to the evaluation of machine learning models. Especially in high-stakes applications where labeled target data are scarce, evaluation of models' generalization performance under distribution shift is a pressing need. We focus on two practical scenarios: (1) Before deployment, how to select the best model for unlabeled target data? (2) After deployment, how to monitor model performance under distribution shift? The central need in both cases is a reliable and label-free proxy metric. Yet existing proxy metrics, such as model confidence or accuracy-on-the-line, are often unreliable as they only assess model output while ignoring the internal mechanisms that produce them. We address this limitation by introducing a new perspective: using the inner workings of a model, i.e., circuits, as a predictive metric of generalization performance. Leveraging circuit discovery, we extract the causal interactions between internal representations as a circuit, from which we derive two metrics tailored to the two practical scenarios. (1) Before deployment, we introduce Dependency Depth Bias, which measures different models' generalization capability on target data. (2) After deployment, we propose Circuit Shift Score, which predicts a model's generalization under different distribution shifts. Across various tasks, both metrics demonstrate significantly improved correlation with generalization performance, outperforming existing proxies by an average of 13.4\% and 34.1\%, respectively. Our code is available at https://github.com/deep-real/GenCircuit.

Related papers

• Causal Representation Learning for Generalisable Recommendation [1.8124328823188354]
  Predictive models trained on observational data often fail to generalise to the distributions they encounter when deployed.<n>We propose an information-theoretic disentanglement criterion and prove that its optimum depends only on the causal components of the input.<n>Our headline evaluation is an A/B test with millions of users on Spotify, applied to a production ranker for personalised playlist generation.
  arXiv  Detail & Related papers  (2026-05-26T13:58:36Z)
• Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Ranking [46.95596181965493]
  This paper presents a unified and practical framework for unsupervised model evaluation and ranking.<n>We show that hybrid metrics consistently outperform single-aspect metrics on both dataset-centric and model-centric evaluation settings.
  arXiv  Detail & Related papers  (2025-10-03T12:48:11Z)
• Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMs [63.82840470917859]
  We show that the decoding mechanism of dLLMs can be used as a powerful tool for model attribution.<n>We propose a novel information extraction scheme called the Directed Decoding Map (DDM), which captures structural relationships between decoding steps and better reveals model-specific behaviors.
  arXiv  Detail & Related papers  (2025-10-02T06:25:10Z)
• Quantifying Uncertainty and Variability in Machine Learning: Confidence Intervals for Quantiles in Performance Metric Distributions [0.17265013728931003]
  Machine learning models are widely used in applications where reliability and robustness are critical.<n>Model evaluation often relies on single-point estimates of performance metrics that fail to capture the inherent variability in model performance.<n>This contribution explores the use of quantiles and confidence intervals to analyze such distributions, providing a more complete understanding of model performance and its uncertainty.
  arXiv  Detail & Related papers  (2025-01-28T13:21:34Z)
• Yesterday's News: Benchmarking Multi-Dimensional Out-of-Distribution Generalization of Misinformation Detection Models [16.581487607358238]
  This article introduces misinfo-general, a benchmark dataset for evaluating misinformation models' ability to perform out-of-distribution generalization.<n>We identify time, event, topic, publisher, political bias, misinformation type as important axes for generalization.<n>We show how this model fails desiderata, which is not necessarily obvious from classification metrics.
  arXiv  Detail & Related papers  (2024-10-12T09:46:36Z)
• An Offline Metric for the Debiasedness of Click Models [52.25681483524383]
  Click models are a common method for extracting information from user clicks.<n>Recent work shows that the current evaluation practices in the community fail to guarantee that a well-performing click model generalizes well to downstream tasks.<n>We introduce the concept of debiasedness in click modeling and derive a metric for measuring it.
  arXiv  Detail & Related papers  (2023-04-19T10:59:34Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Evaluating Predictive Uncertainty and Robustness to Distributional Shift Using Real World Data [0.0]
  We propose metrics for general regression tasks using the Shifts Weather Prediction dataset. We also present an evaluation of the baseline methods using these metrics.
  arXiv  Detail & Related papers  (2021-11-08T17:32:10Z)
• Estimating Generalization under Distribution Shifts via Domain-Invariant Representations [75.74928159249225]
  We use a set of domain-invariant predictors as a proxy for the unknown, true target labels. The error of the resulting risk estimate depends on the target risk of the proxy model.
  arXiv  Detail & Related papers  (2020-07-06T17:21:24Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.