Tailoring Strictly Proper Scoring Rules for Downstream Tasks: An Application to Causal Inference

• URL: http://arxiv.org/abs/2606.03332v1
• Date: Tue, 02 Jun 2026 08:41:25 GMT
• Title: Tailoring Strictly Proper Scoring Rules for Downstream Tasks: An Application to Causal Inference
• Abstract summary: We propose a principled framework for deriving task-specific strictly proper scoring rules by matching the local curvature of the downstream error metric.<n>We apply this to the Average Treatment Effect (ATE) estimation, deriving a closed-form loss and its corresponding canonical probability mapping.
• Score: 48.78219918881965
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Probabilistic models are typically trained using task-agnostic objectives like log-loss, which can lead to significant errors in downstream estimation. This disconnect is especially critical in Inverse Probability Weighting (IPW) for causal inference, where propensity score errors near $0$ and $1$ often lead to high bias and variance. We propose a principled framework for deriving task-specific strictly proper scoring rules by matching the local curvature of the downstream error metric. We apply this to the Average Treatment Effect (ATE) estimation, deriving a closed-form loss and its corresponding canonical probability mapping that can be readily integrated with any model like a neural network or a gradient boosting algorithm. Extensive evaluations on causal inference benchmarks demonstrate that our tailored objective consistently outperforms standard likelihood-based and covariate-balancing approaches.

Related papers

• Generalized Score Matching for Parameter Estimation on Convex Domains [13.954816561132942]
  We derive the generalized score matching objective on a convex subset of $mathbbRd$.<n>We show how classical score matching as well as domain-adapted variants for non-negative data arise naturally within the proposed framework.<n>Our derivation sheds new light on the scope and applicability of generalized score matching in various problem settings.
  arXiv  Detail & Related papers  (2026-09-10T13:23:41Z)
• A Perception vs. Distortion Perspective on Score-Based Generative Channel Estimation [2.378977673869692]
  We present a theoretically grounded interpretation of score-based channel estimation.<n>We quantify the excess risk incurred by standard distortion-minimization approaches.<n>Under high predictive uncertainty, the large excess risk gap can be offset by score-based estimation.
  arXiv  Detail & Related papers  (2026-06-15T15:00:24Z)
• Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration [76.08899010904652]
  CapCal is a training-free framework that mechanically decouples positional bias from ranking decisions.<n>It achieves superior performance among training-free methods while preserving single-pass efficiency.
  arXiv  Detail & Related papers  (2026-04-11T10:47:22Z)
• A Novel Framework for Uncertainty Quantification via Proper Scores for Classification and Beyond [1.5229257192293202]
  We propose a novel framework for uncertainty quantification in machine learning, which is based on proper scores.<n>Specifically, we use the kernel score, a kernel-based proper score, for evaluating sample-based generative models.<n>We generalize the calibration-sharpness decomposition beyond classification, which motivates the definition of proper calibration errors.
  arXiv  Detail & Related papers  (2025-08-25T13:11:03Z)
• Typicalness-Aware Learning for Failure Detection [26.23185979968123]
  Deep neural networks (DNNs) often suffer from the overconfidence issue, where incorrect predictions are made with high confidence scores. We propose a novel approach called Typicalness-Aware Learning (TAL) to address this issue and improve failure detection performance.
  arXiv  Detail & Related papers  (2024-11-04T11:09:47Z)
• PACMAN: PAC-style bounds accounting for the Mismatch between Accuracy and Negative log-loss [28.166066663983674]
  The ultimate performance of machine learning algorithms for classification tasks is usually measured in terms of the empirical error probability (or accuracy) based on a testing dataset. For classification tasks, this loss function is often the negative log-loss that leads to the well-known cross-entropy risk. We introduce an analysis based on point-wise PAC approach over the generalization gap considering the mismatch of testing based on the accuracy metric and training on the negative log-loss.
  arXiv  Detail & Related papers  (2021-12-10T14:00:22Z)
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality [106.50867011164584]
  We show that we can compute the exact Bayes error of generative models learned using normalizing flows. We use our approach to conduct a thorough investigation of state-of-the-art classification models.
  arXiv  Detail & Related papers  (2021-06-07T06:21:20Z)
• Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap [140.98628848491146]
  We introduce deconfounding scores, which induce better overlap without biasing the target of estimation. We show that deconfounding scores satisfy a zero-covariance condition that is identifiable in observed data. In particular, we show that this technique could be an attractive alternative to standard regularizations.
  arXiv  Detail & Related papers  (2021-04-12T18:50:11Z)
• Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation [99.92568326314667]
  We propose the amortized conditional normalized maximum likelihood (ACNML) method as a scalable general-purpose approach for uncertainty estimation. Our algorithm builds on the conditional normalized maximum likelihood (CNML) coding scheme, which has minimax optimal properties according to the minimum description length principle. We demonstrate that ACNML compares favorably to a number of prior techniques for uncertainty estimation in terms of calibration on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2020-11-05T08:04:34Z)
• Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited [68.8204255655161]
  We introduce the CORP approach, which generates provably statistically Consistent, Optimally binned, and Reproducible reliability diagrams in an automated way. Corpor is based on non-parametric isotonic regression and implemented via the Pool-adjacent-violators (PAV) algorithm.
  arXiv  Detail & Related papers  (2020-08-07T08:22:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.