Bridging Domain Expertise and Generalization for Performance Estimation

• URL: http://arxiv.org/abs/2606.06335v1
• Date: Thu, 04 Jun 2026 16:10:04 GMT
• Title: Bridging Domain Expertise and Generalization for Performance Estimation
• Abstract summary: Performance estimation under distribution shift aims to predict how a model behaves on an unlabeled test set whose distribution differs from the training data.<n>Existing approaches rely solely on the outputs of the given model whose biases are amplified once the distribution shifts.<n>We propose Fused Reference Alignment Prediction (FRAP) to construct a more reliable surrogate of the ground-truth labels.
• Score: 44.139948614374106
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Performance estimation under distribution shift aims to predict how a model behaves on an unlabeled test set whose distribution differs from the training data, a scenario that requires reliable indicators that can faithfully reflect model behavior without ground-truth labels. Existing approaches rely solely on the outputs of the given model whose biases are amplified once the distribution shifts, weakening the correlation with the true performance. Motivated by this limitation, we propose Fused Reference Alignment Prediction (FRAP), which leverages the complementary strengths of an external foundation model and the base model to construct a more reliable surrogate of the ground-truth labels. FRAP aligns the prediction distribution of the foundation model with that of the base model by applying temperature-scaled calibration that minimizes their divergence. The aligned predictions are fused through confidence-based weighting into a refined reference distribution that integrates robustness from the foundation model and domain-specific expertise from the base model, and performance estimation is obtained by measuring how closely the base model predictions agree with this reference. Extensive experiments across diverse datasets and architectures show that FRAP provides consistent and substantial improvements over representative performance-estimation methods under distribution shift.

Related papers

• Joint Model and Data Sparsification via the Marginal Likelihood [53.29070892356214]
  We propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification.<n>This symmetric pruning of model and data offers a natural extension that preserves conjugacy.<n> Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.
  arXiv  Detail & Related papers  (2026-05-28T13:26:53Z)
• On Predicting the Post-training Potential of Pre-trained LLMs [60.07459271263409]
  We introduce a new task of predicting post-training potential - forecasting a base model's performance before post-training.<n>We propose RuDE, a unified framework that bypasses the generation gap of base models by leveraging response discrimination.<n>Experiments demonstrate a correlation greater than 90% with post-training performance.
  arXiv  Detail & Related papers  (2026-05-12T11:33:49Z)
• Beyond the Mean: Modelling Annotation Distributions in Continuous Affect Prediction [3.3187704612685267]
  We propose a distribution-aware framework that models annotation consensus using the Beta distribution.<n>Instead of predicting a single affect value, models estimate the mean and standard deviation of the annotation distribution.<n>As a result, the model captures not only the central tendency of emotional perception but also variability, asymmetry, and uncertainty in annotator responses.
  arXiv  Detail & Related papers  (2026-04-08T15:26:56Z)
• STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction [78.0692157478247]
  We propose STAR, a framework that bridges data-driven STatistical expectations with knowledge-driven Agentic Reasoning.<n>We show that STAR consistently outperforms all baselines on both score-based and rank-based metrics.
  arXiv  Detail & Related papers  (2026-02-12T16:30:07Z)
• Nonparametric Data Attribution for Diffusion Models [57.820618036556084]
  Data attribution for generative models seeks to quantify the influence of individual training examples on model outputs.<n>We propose a nonparametric attribution method that operates entirely on data, measuring influence via patch-level similarity between generated and training images.
  arXiv  Detail & Related papers  (2025-10-16T03:37:16Z)
• Prior Distribution and Model Confidence [0.0]
  We propose a framework to understand the confidence of model predictions on unseen data without the need for retraining.<n>Our approach filters out low-confidence predictions based on their distance from the training distribution in the embedding space.<n>The proposed method is model-agnostic and generalizable, with potential applications beyond computer vision.
  arXiv  Detail & Related papers  (2025-09-05T20:17:26Z)
• On conditional diffusion models for PDE simulations [53.01911265639582]
  We study score-based diffusion models for forecasting and assimilation of sparse observations. We propose an autoregressive sampling approach that significantly improves performance in forecasting. We also propose a new training strategy for conditional score-based models that achieves stable performance over a range of history lengths.
  arXiv  Detail & Related papers  (2024-10-21T18:31:04Z)
• Quantifying Distribution Shifts and Uncertainties for Enhanced Model Robustness in Machine Learning Applications [0.0]
  This study explores model adaptation and generalization by utilizing synthetic data. We employ quantitative measures such as Kullback-Leibler divergence, Jensen-Shannon distance, and Mahalanobis distance to assess data similarity. Our findings suggest that utilizing statistical measures, such as the Mahalanobis distance, to determine whether model predictions fall within the low-error "interpolation regime" or the high-error "extrapolation regime" provides a complementary method for assessing distribution shift and model uncertainty.
  arXiv  Detail & Related papers  (2024-05-03T10:05:31Z)
• Enhancing Robustness of Foundation Model Representations under Provenance-related Distribution Shifts [8.298173603769063]
  We examine the stability of models based on foundation models under distribution shift. We focus on confounding by provenance, a form of distribution shift that emerges in the context of multi-institutional datasets. Results indicate that while foundation models do show some out-of-the-box robustness to confounding-by-provenance related distribution shifts, this can be improved through adjustment.
  arXiv  Detail & Related papers  (2023-12-09T02:02:45Z)
• Guide the Learner: Controlling Product of Experts Debiasing Method Based on Token Attribution Similarities [17.082695183953486]
  A popular workaround is to train a robust model by re-weighting training examples based on a secondary biased model. Here, the underlying assumption is that the biased model resorts to shortcut features. We introduce a fine-tuning strategy that incorporates the similarity between the main and biased model attribution scores in a Product of Experts loss function.
  arXiv  Detail & Related papers  (2023-02-06T15:21:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.