UpliftBench: Revealing Outcome-Regime and Objective Mismatch in Uplift Evaluation

• URL: http://arxiv.org/abs/2608.00915v1
• Date: Sun, 02 Aug 2026 00:51:44 GMT
• Title: UpliftBench: Revealing Outcome-Regime and Objective Mismatch in Uplift Evaluation
• Abstract summary: UpliftBench evaluates 12 uplift estimators under an outer-test-isolated, multi-objective protocol.<n>We show the disagreement is substantially about metrics, not models.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Uplift modeling (conditional-average-treatment-effect estimation) drives personalized targeting, yet published uplift benchmarks frequently disagree on which estimator performs best; we show the disagreement is substantially about metrics, not models. UpliftBench evaluates 12 uplift estimators under an outer-test-isolated, multi-objective protocol across seven dataset families; its two findings are identified where a reference objective exists -- F1 on the standard continuous benchmark (IHDP), F2 in a within-sample case study on Jobs. On that benchmark, Qini shows no detectable alignment with effect accuracy -- across all 100 IHDP realizations its mean rank correlation with effect accuracy is +0.07, 95% CI [-0.03, +0.16] -- while AUUC is consistently more aligned (paired prefix-mean-AUUC-over-Qini gap +0.49 [+0.40, +0.59]; the shipped cumulative-gain AUUC aligns better still, +0.73). On Jobs, ranking metrics are structurally insufficient for a sign-threshold policy because they discard the score level; empirically, within the released split-rotation analysis direct policy-risk selection yields lower benchmark regret than random model selection while Qini, AUUC, and uplift-at-$k$ do not (14-15% regret). Calibrating the decision threshold removes 81% of the Qini-selection regret. Both findings are bounded, not universal: F1 is not detected on either validation family (the ACIC and Revenue-Synthetic gaps are both indistinguishable from zero), and F2 vanishes under a budgeted-value objective where rank suffices. UpliftBench releases versioned loaders, fixed protocols, result artifacts, and a reproducible living leaderboard; the public repository accompanies the paper.

Related papers

• Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay [0.6768558752130311]
  We audit against policy-conditional ground truth from executed replay in a single-agent tool environment.<n>No step-level credit signals we audit show reliable incremental fidelity beyond marginal-matched shuffled control.<n>A confidence-only router recovers pivotal steps at chance level, but cuts judge cost by 13.1% per turn.
  arXiv  Detail & Related papers  (2026-08-20T08:04:00Z)
• Decomposing Wrong-Consensus Agreement in LLM Self-Consistency: A GPT-4.1 Case Study [1.2716872085463884]
  Majority voting over multiple LLM samples is widely used to raise answer accuracy, yet its gain varies erratically.<n>This paper gives a quantitative account of this failure.
  arXiv  Detail & Related papers  (2026-08-19T10:50:15Z)
• Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation [2.1734195143282693]
  We propose reliability@k, the same estimator applied correctly, with n = independent rollouts and c = fully-passing rollouts per (task, agent) pair.<n>Motivated by evidence that functional correctness does not imply security safety, we additionally propose security-adjusted reliability@k.
  arXiv  Detail & Related papers  (2026-08-11T16:01:15Z)
• Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation [0.0]
  Agentic AI systems are evaluated using automated benchmarks whose scores justify deployment decisions, safety certifications, and regulatory compliance claims.<n>We present an empirical analysis demonstrating that these scores are systematically less trustworthy than current practice acknowledges.
  arXiv  Detail & Related papers  (2026-08-01T17:50:12Z)
• Beyond Point Estimates: Benchmarking Uncertainty Quantification Methods on the AION-1 Astronomical Foundation Model [47.71489969727656]
  Conformalized Quantile Regression (CQR) delivers the best coverage in the bin with the poorest model predictions.<n>Only the Locally Valid and Discriminative (LVD) framework provides finite-sample emphlocal validity.
  arXiv  Detail & Related papers  (2026-06-05T18:29:19Z)
• A Pre-Registered Causal Partition of Self-Consistency Elicitation and Reward Design in RLVR [1.2958054117511815]
  Reinforcement learning from rewards improves reasoning even when the reward signal is spurious.<n> Practitioners commonly interpret naive = acc(TRUE) - acc(R) as the reward-design effect.<n>We prove this estimand is systematically biased.
  arXiv  Detail & Related papers  (2026-06-04T09:35:54Z)
• Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score [14.579552536669217]
  Unlearning in Vision-Language Models (VLMs) is required for compliance with the General Data Protection Regulation Code.<n>We present the first study of reliability in multimodal unlearning.<n>We introduce the Unified Quality Score (UQS), a metric with weights derived from each metric's Spearman correlation with the metric distance.
  arXiv  Detail & Related papers  (2026-05-04T04:13:00Z)
• Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classification [43.86209535250233]
  Cross-prompt variance is strongly associated with prompt-level disagreement and higher error.<n>A training-free mean ensemble improves NLL on all 14 dataset-model evaluation pairs.<n>We frame this as a reliability stress test for zero-shot VLM first-token safety scores.
  arXiv  Detail & Related papers  (2026-05-01T01:06:30Z)
• Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment [53.72927532626824]
  We show that subsets of just 50 examples (0.3% of data) can achieve over 0.93 Pearson correlation with full benchmark scores.<n>To better predict preferences, we trained regression models on these selected subsets, achieving 0.98 correlation.<n>This demonstrates that in regression modeling, well-curated subsets outpredict the full benchmark, showing quality over quantity.
  arXiv  Detail & Related papers  (2026-04-20T00:57:31Z)
• Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration [76.08899010904652]
  CapCal is a training-free framework that mechanically decouples positional bias from ranking decisions.<n>It achieves superior performance among training-free methods while preserving single-pass efficiency.
  arXiv  Detail & Related papers  (2026-04-11T10:47:22Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Eliminating Position Bias of Language Models: A Mechanistic Approach [119.34143323054143]
  Position bias has proven to be a prevalent issue of modern language models (LMs)<n>Our mechanistic analysis attributes the position bias to two components employed in nearly all state-of-the-art LMs: causal attention and relative positional encodings.<n>By eliminating position bias, models achieve better performance and reliability in downstream tasks, including LM-as-a-judge, retrieval-augmented QA, molecule generation, and math reasoning.
  arXiv  Detail & Related papers  (2024-07-01T09:06:57Z)
• Optimizing Partial Area Under the Top-k Curve: Theory and Practice [151.5072746015253]
  We develop a novel metric named partial Area Under the top-k Curve (AUTKC) AUTKC has a better discrimination ability, and its Bayes optimal score function could give a correct top-K ranking with respect to the conditional probability. We present an empirical surrogate risk minimization framework to optimize the proposed metric.
  arXiv  Detail & Related papers  (2022-09-03T11:09:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.