Can You Trust Frozen Hematology Foundation Models under Acquisition Shift?

• URL: http://arxiv.org/abs/2608.25148v1
• Date: Tue, 25 Aug 2026 20:55:57 GMT
• Title: Can You Trust Frozen Hematology Foundation Models under Acquisition Shift?
• Abstract summary: We audit 15 frozen encoders across four public single-cell acquisition domains.<n>In-domain linear-probe macro-F1 is saturated (0.98-0.997), yet cross-dataset macro-F1 drops 34-72%.<n>DinoBloom-L, the in-domain best, falls to 10th of 15 on the most-shifted target (MLL23), behind RedDino and several general and pathology encoders.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Frozen hematology foundation-model (FM) embeddings reach near-saturated in-domain white-blood-cell (WBC) accuracy, but clinical deployment demands reliability across scanners, sites, stains and preparation pipelines. We audit 15 frozen encoders (hematology, pathology, and general vision) across four public single-cell acquisition domains along two axes: accuracy robustness and calibration. In-domain linear-probe macro-F1 is saturated (0.98-0.997), yet cross-dataset macro-F1 drops 34-72% and rankings re-order: DinoBloom-L, the in-domain best, falls to 10th of 15 on the most-shifted target (MLL23) at the benchmark's shared 224-px input, behind RedDino and several general and pathology encoders. Rank transfer is probe-dependent: 1-NN retrieval is more stable on average than a source-fitted linear head (median $ρ$ 0.65 vs 0.45), but neither probe universally predicts target robustness. Calibration also collapses: source-trained probes are nearly calibrated in-domain (expected calibration error, ECE, 0.004) but confidently wrong off-domain (ECE 0.35), and source-fitted temperature scaling transfers poorly. We further audit pretraining exposure and identify MLL23 as DinoBloom's internal cohort; because DinoBloom's only held-out dataset is also our source domain, this benchmark cannot isolate exposure from scanner-associated shift. Label-free adaptation and marginal-entropy-based model selection appear safe under balanced evaluation but fail under realistic WBC class-prior shift. Class-Balanced Re-standardization (CBR), a training-free pseudo-label-balanced feature normalization, improves all evaluated target-prior scenario means and partially improves calibration, although encoder-level exceptions and residual miscalibration remain. Hematology FM benchmarks must therefore jointly audit accuracy, calibration, exposure, and class-prior robustness.

Related papers

• C-Score: Beyond Accuracy for Robustness Assessment in Semi-Supervised Learning under Open-World Unlabeled Contamination [6.334765412132314]
  Pseudo-label-based semi-supervised learning has achieved strong performance due to its simplicity and scalability.<n>In practical deployment, unlabeled data are often collected from open environments and may contain OOD samples.<n>Under such contamination, OOD samples may still receive high-confidence predictions and be incorporated into training as if they were valid target examples.<n>This creates an important evaluation problem: clean in-distribution test accuracy may appear stable even when the internal learning dynamics of SSL have already deteriorated.
  arXiv  Detail & Related papers  (2026-08-21T01:58:39Z)
• Beyond Accuracy: Assessing Calibration of Geospatial Foundation Models and Their Sensitivity to Distribution Shifts [15.070040506693738]
  We show that the promised deployment in critical EO tasks requires further angles of analysis.<n> EO-pretrained encoders move less under corruption while losing just as much task information.<n>We find that confidence-based abstention cannot defer around confidently wrong predictions.
  arXiv  Detail & Related papers  (2026-08-17T14:16:14Z)
• Calibrated Hybrid CNN-Transformer for Retinal OCT Classification [1.0817607291351121]
  Deep models for retinal optical coherence tomography ( OCT) classification report high accuracy but rarely report whether their confidence can be trusted.<n>We pair a hybrid convolutional-Transformer encoder with a gradient-boosting (XGBoost) classification head and a three-part clinical safety layer: confidence calibration, out-of-distribution (OOD) rejection, and per-prediction uncertainty flagging.<n>On four-class OCT (84,495 scans) the model reaches 95.4% accuracy while cutting calibration error twelve-fold (expected calibration error, ECE = 0.0024), so the confidence it reports tracks its true accuracy.
  arXiv  Detail & Related papers  (2026-07-09T23:23:56Z)
• Calibration, Uncertainty Communication, and Deployment Readiness in CKD Risk Prediction: A Framework Evaluation Study [0.0]
  Machine learning models for chronic kidney disease often post strong discrimination scores on internal test sets.<n>We trained five classifiers on the UCI CKD dataset (400 patients, 62.5% CKD prevalence)<n>We evaluated each across calibration quality, conformal prediction coverage, and an eight-criterion deployment readiness framework.
  arXiv  Detail & Related papers  (2026-05-20T17:12:48Z)
• Discovery of Hidden Miscalibration Regimes [52.452902154360565]
  Models may be systematically overconfident on some kinds of inputs and underconfident on others.<n>We define the corresponding miscalibration field and propose a diagnostic framework for estimating it.<n>Our approach learns a calibration-aware representation of the input space and estimates signed local miscalibration by kernel smoothing in the learned geometry.
  arXiv  Detail & Related papers  (2026-05-13T13:07:50Z)
• Signal Fidelity Index-Aware Calibration for Dementia Predictions Across Heterogeneous Real-World Data [1.741250583668341]
  We develop a Signal Fidelity Index (SFI) diagnostic data quality at the patient level in dementia.<n>We test SFI-aware calibration for improving model performance across heterogeneous datasets without outcome labels.
  arXiv  Detail & Related papers  (2025-09-10T15:19:04Z)
• Model Calibration in Dense Classification with Adaptive Label Perturbation [44.62722402349157]
  Existing dense binary classification models are prone to being over-confident. We propose Adaptive Label Perturbation (ASLP) which learns a unique label perturbation level for each training image. ASLP can significantly improve calibration degrees of dense binary classification models on both in-distribution and out-of-distribution data.
  arXiv  Detail & Related papers  (2023-07-25T14:40:11Z)
• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• Sample-dependent Adaptive Temperature Scaling for Improved Calibration [95.7477042886242]
  Post-hoc approach to compensate for neural networks being wrong is to perform temperature scaling. We propose to predict a different temperature value for each input, allowing us to adjust the mismatch between confidence and accuracy. We test our method on the ResNet50 and WideResNet28-10 architectures using the CIFAR10/100 and Tiny-ImageNet datasets.
  arXiv  Detail & Related papers  (2022-07-13T14:13:49Z)
• Privacy Preserving Recalibration under Domain Shift [119.21243107946555]
  We introduce a framework that abstracts out the properties of recalibration problems under differential privacy constraints. We also design a novel recalibration algorithm, accuracy temperature scaling, that outperforms prior work on private datasets.
  arXiv  Detail & Related papers  (2020-08-21T18:43:37Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.