Label-Efficient Monitoring of Classification Models via Stratified Importance Sampling

• URL: http://arxiv.org/abs/2601.22326v1
• Date: Thu, 29 Jan 2026 21:12:00 GMT
• Title: Label-Efficient Monitoring of Classification Models via Stratified Importance Sampling
• Abstract summary: Stratified Importance Sampling (SIS) is a principled, label-efficient, and operationally lightweight methodology for post-deployment model monitoring.<n> SIS yields unbiased estimators with strict finite-sample mean squared error (MSE) improvements over both importance sampling (IS) and stratified random sampling (SRS)<n>Experiments across binary and multiclass tasks demonstrate consistent efficiency improvements under fixed label budgets.
• Score: 0.3222802562733787
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Monitoring the performance of classification models in production is critical yet challenging due to strict labeling budgets, one-shot batch acquisition of labels and extremely low error rates. We propose a general framework based on Stratified Importance Sampling (SIS) that directly addresses these constraints in model monitoring. While SIS has previously been applied in specialized domains, our theoretical analysis establishes its broad applicability to the monitoring of classification models. Under mild conditions, SIS yields unbiased estimators with strict finite-sample mean squared error (MSE) improvements over both importance sampling (IS) and stratified random sampling (SRS). The framework does not rely on optimally defined proposal distributions or strata: even with noisy proxies and sub-optimal stratification, SIS can improve estimator efficiency compared to IS or SRS individually, though extreme proposal mismatch may limit these gains. Experiments across binary and multiclass tasks demonstrate consistent efficiency improvements under fixed label budgets, underscoring SIS as a principled, label-efficient, and operationally lightweight methodology for post-deployment model monitoring.

Related papers

• Diachronic Sample Integration: Robust Tail-Risk Estimation with Generative Models [17.40627902498753]
  We introduce Diachronic Sample Integration (DSI), a test-time inference framework that ensembles generated samples across checkpoints from a training trajectory.<n>DSI targets a checkpoint-mixture distribution that averages checkpoint-specific tail fluctuations rather than relying on a single brittle endpoint.<n>It substantially reduces tail-estimation error compared to single-checkpoint baselines under fixed simulation budgets, outperforming standard diffusion and state-of-the-art tail-aware baselines without modifying the generative objective.
  arXiv  Detail & Related papers  (2026-07-12T15:45:23Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction [78.0692157478247]
  We propose STAR, a framework that bridges data-driven STatistical expectations with knowledge-driven Agentic Reasoning.<n>We show that STAR consistently outperforms all baselines on both score-based and rank-based metrics.
  arXiv  Detail & Related papers  (2026-02-12T16:30:07Z)
• A Unified and Stable Risk Minimization Framework for Weakly Supervised Learning with Theoretical Guarantees [33.15955234458642]
  Weakly supervised learning has emerged as a practical alternative to fully supervised learning when complete and accurate labels are costly or infeasible to acquire.<n>We propose a principled, unified framework that bypasses such post-hoc adjustments by formulating a stable surrogate risk grounded in the structure of weakly supervised data.
  arXiv  Detail & Related papers  (2025-11-28T00:57:04Z)
• Labels Matter More Than Models: Quantifying the Benefit of Supervised Time Series Anomaly Detection [56.302586730134806]
  Time series anomaly detection (TSAD) is a critical data mining task often constrained by label scarcity.<n>Current research predominantly focuses on Unsupervised Time-series Anomaly Detection.<n>This paper challenges the premise that architectural complexity is the optimal path for TSAD.
  arXiv  Detail & Related papers  (2025-11-20T08:32:49Z)
• Additive Distributionally Robust Ranking and Selection [0.8283940114367679]
  We propose a simple additive allocation (AA) procedure that aims to exclusively sample the $k + m - 1$ previously hypothesized critical scenarios.<n>We prove that AA is consistent and, surprisingly, achieves additivity in the strongest sense.<n>Results offer new and counterintuitive insights into the additive structure of DRR&S.
  arXiv  Detail & Related papers  (2025-09-07T17:36:29Z)
• ConformalSAM: Unlocking the Potential of Foundational Segmentation Models in Semi-Supervised Semantic Segmentation with Conformal Prediction [57.930531826380836]
  This work explores whether a foundational segmentation model can address label scarcity in the pixel-level vision task as an annotator for unlabeled images.<n>We propose ConformalSAM, a novel SSSS framework which first calibrates the foundation model using the target domain's labeled data and then filters out unreliable pixel labels of unlabeled data.
  arXiv  Detail & Related papers  (2025-07-21T17:02:57Z)
• NDCG-Consistent Softmax Approximation with Accelerated Convergence [67.10365329542365]
  We propose novel loss formulations that align directly with ranking metrics.<n>We integrate the proposed RG losses with the highly efficient Alternating Least Squares (ALS) optimization method.<n> Empirical evaluations on real-world datasets demonstrate that our approach achieves comparable or superior ranking performance.
  arXiv  Detail & Related papers  (2025-06-11T06:59:17Z)
• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models [69.798277882245]
  We introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance large language models' reasoning efficiency.<n>UPFT removes the need for labeled data or exhaustive sampling.<n> Experiments show that UPFT matches the performance of supervised methods.
  arXiv  Detail & Related papers  (2025-03-04T18:56:03Z)
• Statistical Properties of Robust Satisficing [5.0139295307605325]
  The Robust Satisficing (RS) model is an emerging approach to robust optimization. This paper comprehensively analyzes the theoretical properties of the RS model. Our experiments show that the RS model consistently outperforms the baseline empirical risk in small-sample regimes.
  arXiv  Detail & Related papers  (2024-05-30T19:57:28Z)
• The Decaying Missing-at-Random Framework: Model Doubly Robust Causal Inference with Partially Labeled Data [8.916614661563893]
  We introduce a missing-at-random (decaying MAR) framework and associated approaches for doubly robust causal inference.<n>This simultaneously addresses selection bias in the labeling mechanism and the extreme imbalance between labeled and unlabeled groups.<n>To ensure robust causal conclusions, we propose a bias-reduced SS estimator for the average treatment effect.
  arXiv  Detail & Related papers  (2023-05-22T07:37:12Z)
• Rethinking Sampling Strategies for Unsupervised Person Re-identification [59.47536050785886]
  We analyze the reasons for the performance differences between various sampling strategies under the same framework and loss function.<n>Group sampling is proposed, which gathers samples from the same class into groups.<n>Experiments on Market-1501, DukeMTMC-reID and MSMT17 show that group sampling achieves performance comparable to state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-07-07T05:39:58Z)
• WSSOD: A New Pipeline for Weakly- and Semi-Supervised Object Detection [75.80075054706079]
  We propose a weakly- and semi-supervised object detection framework (WSSOD) An agent detector is first trained on a joint dataset and then used to predict pseudo bounding boxes on weakly-annotated images. The proposed framework demonstrates remarkable performance on PASCAL-VOC and MSCOCO benchmark, achieving a high performance comparable to those obtained in fully-supervised settings.
  arXiv  Detail & Related papers  (2021-05-21T11:58:50Z)
• Needle in a Haystack: Label-Efficient Evaluation under Extreme Class Imbalance [20.491690754953943]
  This paper develops a framework for online evaluation based on adaptive importance sampling. Experiments demonstrate an average MSE superior to state-of-the-art on fixed label budgets.
  arXiv  Detail & Related papers  (2020-06-12T06:17:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.