Scalable Model-Assisted Multi-Target Estimation in Large Image Collections

• URL: http://arxiv.org/abs/2607.17581v1
• Date: Mon, 20 Jul 2026 05:54:36 GMT
• Title: Scalable Model-Assisted Multi-Target Estimation in Large Image Collections
• Abstract summary: We study the problem of multi-target estimation, where many quantities must be estimated simultaneously.<n>Our framework effectively combines biased model predictions and limited human labels into rigorous scientific measurements.
• Score: 21.336432095392954
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Computer vision models are increasingly used as measurement tools to estimate population-level quantities from large image collections, but prediction errors introduce bias and the resulting estimates lack statistical guarantees required in scientific applications. Prior work uses a Monte Carlo framework to combine model predictions with ground-truth annotations by sampling some images for humans to label and is able to provide unbiased estimates with controllable accuracy, but primarily addresses single-scalar estimation. We study the more general problem of multi-target estimation, where many quantities (e.g., class counts or proportions) must be estimated simultaneously, and adapt sampling and estimation strategies from survey sampling to this setting. Evaluations on five detection and segmentation datasets with 7-80 classes show that importance sampling excels with moderate annotation budgets or fewer targets, whereas uniform sampling with control variates is superior when estimating many targets or operating with minimal labels. Additionally, a subset-based ratio estimator remains highly competitive across all regimes. Ultimately, our framework effectively combines biased model predictions and limited human labels into rigorous scientific measurements.

Related papers

• Measuring consistency via ensemble margin and local prediction variability: Auditing decision systems in the presence of predictive multiplicity [4.924900955306918]
  The Rashomon effect is a machine learning phenomenon where equally accurate models produce different predictions for the same inputs.<n>We propose a consistency criterion that combines the ensemble margin with a measure of local prediction variability for each constituent model.<n>Our experiments show that ensembling models from the Rashomon set substantially reduces the risk of incorrect predictions going unchecked.
  arXiv  Detail & Related papers  (2026-09-01T15:22:01Z)
• On Arbitrary Predictions from Equally Valid Models [49.56463611078044]
  Model multiplicity refers to multiple machine learning models that admit conflicting predictions for the same patient.<n>We show that even small ensembles can mitigate/eliminate predictive multiplicity in practice.
  arXiv  Detail & Related papers  (2025-07-25T16:15:59Z)
• QuEst: Enhancing Estimates of Quantile-Based Distributional Measures Using Model Predictions [12.851704083461616]
  We present QuEst, a principled framework to merge observed and imputed data to deliver point estimates.<n> QuEst covers a range of measures, from tail risk (CVaR) to population segments such as quartiles, that are central to fields such as economics, sociology, education, medicine, and more.<n>We extend QuEst to multidimensional metrics, and introduce an additional optimization technique to further reduce variance in this and other hybrid estimators.
  arXiv  Detail & Related papers  (2025-07-07T17:33:18Z)
• Semiparametric conformal prediction [79.6147286161434]
  We construct a conformal prediction set accounting for the joint correlation structure of the vector-valued non-conformity scores.<n>We flexibly estimate the joint cumulative distribution function (CDF) of the scores.<n>Our method yields desired coverage and competitive efficiency on a range of real-world regression problems.
  arXiv  Detail & Related papers  (2024-11-04T14:29:02Z)
• Precise Model Benchmarking with Only a Few Observations [6.092112060364272]
  We propose an empirical Bayes (EB) estimator that balances direct and regression estimates for each subgroup separately. EB consistently provides more precise estimates of the LLM performance compared to the direct and regression approaches.
  arXiv  Detail & Related papers  (2024-10-07T17:26:31Z)
• A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation [17.351089059392674]
  We propose a framework for model evaluation that includes stratification, sampling, and estimation components. We show that stratification via k-means clustering based on accurate predictions of model performance yields efficient estimators. We also find that model-assisted estimators, which leverage predictions of model accuracy on the unlabeled portion of the dataset, are generally more efficient than the traditional estimates.
  arXiv  Detail & Related papers  (2024-06-11T14:49:04Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Test-time Collective Prediction [73.74982509510961]
  Multiple parties in machine learning want to jointly make predictions on future test points. Agents wish to benefit from the collective expertise of the full set of agents, but may not be willing to release their data or model parameters. We explore a decentralized mechanism to make collective predictions at test time, leveraging each agent's pre-trained model.
  arXiv  Detail & Related papers  (2021-06-22T18:29:58Z)
• Model-based metrics: Sample-efficient estimates of predictive model subpopulation performance [11.994417027132807]
  Machine learning models $-$ now commonly developed to screen, diagnose, or predict health conditions are evaluated with a variety of performance metrics. Subpopulation performance metrics are typically computed using only data from that subgroup, resulting in higher variance estimates for smaller groups. We propose using an evaluation model $-$ a model that describes the conditional distribution of the predictive model score $-$ to form model-based metric (MBM) estimates.
  arXiv  Detail & Related papers  (2021-04-25T19:06:34Z)
• Flexible Model Aggregation for Quantile Regression [92.63075261170302]
  Quantile regression is a fundamental problem in statistical learning motivated by a need to quantify uncertainty in predictions. We investigate methods for aggregating any number of conditional quantile models. All of the models we consider in this paper can be fit using modern deep learning toolkits.
  arXiv  Detail & Related papers  (2021-02-26T23:21:16Z)
• Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model [71.9860741092209]
  Clinical researchers often select among and evaluate risk prediction models. Standard metrics calculated from retrospective data are only related to model utility under certain assumptions. When predictions are delivered repeatedly throughout time, the relationship between standard metrics and utility is further complicated.
  arXiv  Detail & Related papers  (2020-06-02T16:26:49Z)
• Diversity inducing Information Bottleneck in Model Ensembles [73.80615604822435]
  In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. Compared to the most competitive baselines, we show significant improvements in classification accuracy, under a shift in the data distribution.
  arXiv  Detail & Related papers  (2020-03-10T03:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.