Beyond Point Estimates: Benchmarking Uncertainty Quantification Methods on the AION-1 Astronomical Foundation Model

• URL: http://arxiv.org/abs/2606.07771v1
• Date: Fri, 05 Jun 2026 18:29:19 GMT
• Title: Beyond Point Estimates: Benchmarking Uncertainty Quantification Methods on the AION-1 Astronomical Foundation Model
• Abstract summary: Conformalized Quantile Regression (CQR) delivers the best coverage in the bin with the poorest model predictions.<n>Only the Locally Valid and Discriminative (LVD) framework provides finite-sample emphlocal validity.
• Score: 47.71489969727656
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models for astronomical surveys offer powerful learned representations that can be transferred to downstream regression tasks such as galaxy property estimation. However, point predictions alone are insufficient for scientific inference; reliable uncertainty quantification (UQ) is essential. We compare seven UQ methods on galaxy property regression using frozen AION-1 foundation-model embeddings, predicting redshift, stellar mass, stellar-population age, gas-phase metallicity, and specific star-formation rate, from Legacy Survey photometry/imaging and DESI spectra, with PROVABGS-derived labels. Distribution-free conformal methods achieve marginal coverage within $\sim$1\,pp of the nominal 90\% across all properties, while non-conformal baselines (Deep Ensembles, MC~Dropout) fail to calibrate reliably. Among conformal approaches, Conformalized Quantile Regression (CQR) delivers the best coverage in the bin with the poorest model predictions. More importantly, only the Locally Valid and Discriminative (LVD) framework -- particularly when operating on AION-1 embeddings -- also provides finite-sample \emph{local validity}, producing intervals that adapt to each galaxy's local prediction difficulty rather than relying on marginal guarantees alone. These results establish conformal prediction, and LVD in particular, as the preferred UQ framework for uncertainty-aware inference on foundation-model embeddings in astrophysics.

Related papers

• Conditional Evaluation of Language Models with Cheap Auxiliary Signals [7.275460635772361]
  Estimating conditional performance profiles from gold labels alone is expensive.<n>We propose LACE (Local Augmented Control-Variate Evaluation), a semi-supervised estimator for conditional LLM evaluation.<n>We empirically evaluate the primary performance-profile estimator on MATH-500, ScienceQA, MMLU, WinoGrande, HellaSwag, TruthfulQA, GSM8K, and ARC.
  arXiv  Detail & Related papers  (2026-08-17T07:44:22Z)
• Attention-Path Fragility as an Uncertainty Signal in Large Language Models [3.5772286398380255]
  We instantiate this as ASMI, a training-free estimator that masks attention heads and measures the BALD mutual information among the resultingworks.<n>On grounded QA an out-of-fold test shows it adds error-predictive information beyond single-pass confidence and entropy.<n>A head-level analysis shows that what tracks this boundary is not the presence of head-level fragility but whether that fragility couples to errors.
  arXiv  Detail & Related papers  (2026-08-11T16:59:02Z)
• Covariance-Boosted Gaussian Processes for Spatiotemporal Irregularities [0.0]
  Motivated by ionospheric modeling for satellite-based augmentation systems, this paper proposes a Covariance-Boosted Gaussian Process (CBGP) framework.<n>CBGP model efficacy and robustness are demonstrated through out-of-sample testing of both simulated and real-world applications.<n>The modeling of an extensive ionospheric storm dataset over South America suggests accurate and reliable means to compute SBAS ionospheric corrections.
  arXiv  Detail & Related papers  (2026-07-25T03:24:04Z)
• Expanding SPHERE-JEPA: A Family of Statistical Regularizers for the Hypersphere [11.087393998848254]
  In Self-Supervised Learning (SSL), preventing representation collapse by explicitly enforcing a uniform distribution on the unit hypersphere has proven to be effective.<n>Current frameworks typically rely on sliced statistical regularizers such as SIGReg (used in LeJEPA) and SUSReg (used in SPHEREJEPA)<n>This injects projection variance into the training gradients, destabilizing optimization, and hindering convergence.<n>In this work, we first show that analytically integrating out these random projections yields a deterministic Maximum Mean Discrepancy (MMD), bypassing the variance of sliced methods.
  arXiv  Detail & Related papers  (2026-06-16T07:10:05Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• Towards Anytime-Valid Statistical Watermarking [63.02116925616554]
  We develop the first e-value-based watermarking framework, Anchored E-Watermarking, that unifies optimal sampling with anytime-valid inference.<n>Our framework can significantly enhance sample efficiency, reducing the average token budget required for detection by 13-15% relative to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-02-19T18:32:26Z)
• Geometry-Aware Uncertainty Quantification via Conformal Prediction on Manifolds [3.2848713528308817]
  We propose a framework that replaces Euclidean residuals with geodesic nonconformity scores and normalizes them by a cross-validated difficulty estimator to handle heteroscedastic noise.<n>The resulting prediction regions, geodesic caps on the sphere, have position-independent area and adapt their size to local prediction difficulty, yielding substantially more uniform conditional coverage than non-adaptive alternatives.
  arXiv  Detail & Related papers  (2026-02-17T21:12:47Z)
• STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction [78.0692157478247]
  We propose STAR, a framework that bridges data-driven STatistical expectations with knowledge-driven Agentic Reasoning.<n>We show that STAR consistently outperforms all baselines on both score-based and rank-based metrics.
  arXiv  Detail & Related papers  (2026-02-12T16:30:07Z)
• CAAL: Confidence-Aware Active Learning for Heteroscedastic Atmospheric Regression [7.951744148676244]
  Quantifying the impacts of air pollution on health and climate relies on key atmospheric particle properties such as toxicity and hygroscopicity.<n>These properties typically require complex observational techniques or expensive particle-resolved numerical simulations.<n>We propose a confidence-aware active learning framework (CAAL) for efficient and robust sample selection.
  arXiv  Detail & Related papers  (2026-02-12T11:09:58Z)
• Geometric Calibration and Neutral Zones for Uncertainty-Aware Multi-Class Classification [0.0]
  This work bridges information geometry and statistical learning, offering formal guarantees for uncertainty-aware classification in applications requiring rigorous validation.<n> Empirical validation on Adeno-Associated Virus classification demonstrates that the two-stage framework captures 72.5% of errors while deferring 34.5% of samples, reducing automated decision error rates from 16.8% to 6.9%.
  arXiv  Detail & Related papers  (2025-11-26T01:29:49Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Spatial Conformal Inference through Localized Quantile Regression [6.992239210938067]
  Conformal prediction offers valid prediction intervals without relying on parametric assumptions.<n>We propose Localized Spatial Conformal Prediction (L SCP), a conformal prediction method designed specifically for spatial data.<n>L SCP achieves accurate coverage with significantly tighter and more consistent prediction intervals compared to existing methods.
  arXiv  Detail & Related papers  (2024-12-02T04:15:06Z)
• Distribution-free uncertainty quantification for classification under label shift [105.27463615756733]
  We focus on uncertainty quantification (UQ) for classification problems via two avenues. We first argue that label shift hurts UQ, by showing degradation in coverage and calibration. We examine these techniques theoretically in a distribution-free framework and demonstrate their excellent practical performance.
  arXiv  Detail & Related papers  (2021-03-04T20:51:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.