Real vs. Semi-Simulated: Rethinking Evaluation for Treatment Effect Estimation

• URL: http://arxiv.org/abs/2605.10430v1
• Date: Mon, 11 May 2026 12:04:02 GMT
• Title: Real vs. Semi-Simulated: Rethinking Evaluation for Treatment Effect Estimation
• Abstract summary: We conduct a large-scale empirical study of treatment effect evaluation across standard semi-simulated benchmark families and real-world datasets.<n>We evaluate these methods using observable metrics common in application-oriented literature, alongside counterfactual metrics commonly used in methods papers.
• Score: 1.286026162904759
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Estimating heterogeneous treatment effects with machine learning has attracted substantial attention in both academic research and industrial practice. However, the two communities often evaluate models under markedly different conditions. Methodological work typically relies on semi-simulated benchmarks and metrics that require counterfactual outcomes, whereas real-world applications rely on observable metrics based on ranking or test outcomes. Despite the well-known gap between methodological progress and practical deployment, the relationship between these evaluation regimes has not been examined systematically. We conduct a large-scale empirical study of treatment effect evaluation across standard semi-simulated benchmark families and real-world datasets. Our benchmark covers meta-learners paired with multiple base learners, as well as specialized causal machine learning models. We evaluate these methods using observable metrics common in application-oriented literature, alongside counterfactual metrics commonly used in methods papers. Our results reveal two complementary gaps. First, counterfactual metrics do not reliably recover the estimators preferred by observable metrics, even on the same semi-simulated benchmarks. Second, rankings obtained on semi-simulated benchmarks do not transfer to real datasets. We further find that simple meta-learners with strong base models are consistently competitive, in contrast to specialized causal models. Overall, our findings suggest that progress in treatment effect estimation research should not be assessed solely through counterfactual metrics and semi-simulated benchmarks, but it would benefit from incorporating observable metrics and real-data validation.

Related papers

• Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation [47.29900218955283]
  Mutual information estimation is a central problem in machine learning and statistics.<n>Existing benchmarks evaluate estimators on simplified, low-dimensional distributions, leaving their performance on complex, realistic data largely unexplored.<n>We address this gap with a comprehensive benchmarking framework grounded in a unified copula-theoretic perspective.
  arXiv  Detail & Related papers  (2026-07-03T16:55:55Z)
• Revisiting OmniAnomaly for Anomaly Detection: performance metrics and comparison with PCA-based models [0.08155575318208629]
  This work revisits OmniAnomaly, a widely used recurrent model for MTSAD, and compares it with a simple linear baseline based on Principal Component Analysis.<n>Both methods are evaluated under thresholding and evaluation procedures, with experiments repeated across 100 runs for each of the 28 machines in the dataset.<n>The results show large variability across machines and show that can achieve performance comparable to OmniAnomaly, and even outperform it when point-adjustment is not applied.
  arXiv  Detail & Related papers  (2026-03-19T14:54:38Z)
• Reference-Specific Unlearning Metrics Can Hide the Truth: A Reality Check [60.77691669644931]
  We propose Functional Alignment for Distributional Equivalence (FADE), a novel metric that measures distributional similarity between unlearned and reference models.<n>We show that FADE captures functional alignment across the entire output distribution, providing a principled assessment of genuine unlearning.<n>These findings expose fundamental gaps in current evaluation practices and demonstrate that FADE provides a more robust foundation for developing and assessing truly effective unlearning methods.
  arXiv  Detail & Related papers  (2025-10-14T20:50:30Z)
• How NOT to benchmark your SITE metric: Beyond Static Leaderboards and Towards Realistic Evaluation [11.33816414982401]
  Transferability estimation metrics are used to find a high-performing pre-trained model for a given target task.<n>Despite the growing interest in developing such metrics, the benchmarks used to measure their progress have gone largely unexamined.<n>We argue that the benchmarks on which these metrics are evaluated are fundamentally flawed.
  arXiv  Detail & Related papers  (2025-10-07T20:38:12Z)
• Testing Generalizability in Causal Inference [3.547529079746247]
  No formal procedure exists for statistically evaluating generalizability in machine learning algorithms.<n>We propose a systematic framework for statistically evaluating the generalizability of high-dimensional causal inference models.
  arXiv  Detail & Related papers  (2024-11-05T11:44:00Z)
• A Large-Scale Neutral Comparison Study of Survival Models on Low-Dimensional Data [7.199059106376138]
  This work presents the first large-scale neutral benchmark experiment focused on single-event, right-censored, low-dimensional survival data. We benchmark 18 models, ranging from classical statistical approaches to many common machine learning methods, on 32 publicly available datasets.
  arXiv  Detail & Related papers  (2024-06-06T14:13:38Z)
• Don't Make Your LLM an Evaluation Benchmark Cheater [142.24553056600627]
  Large language models(LLMs) have greatly advanced the frontiers of artificial intelligence, attaining remarkable improvement in model capacity. To assess the model performance, a typical approach is to construct evaluation benchmarks for measuring the ability level of LLMs. We discuss the potential risk and impact of inappropriately using evaluation benchmarks and misleadingly interpreting the evaluation results.
  arXiv  Detail & Related papers  (2023-11-03T14:59:54Z)
• Systematic Evaluation of Predictive Fairness [60.0947291284978]
  Mitigating bias in training on biased datasets is an important open problem. We examine the performance of various debiasing methods across multiple tasks. We find that data conditions have a strong influence on relative model performance.
  arXiv  Detail & Related papers  (2022-10-17T05:40:13Z)
• Doing Great at Estimating CATE? On the Neglected Assumptions in Benchmark Comparisons of Treatment Effect Estimators [91.3755431537592]
  We show that even in arguably the simplest setting, estimation under ignorability assumptions can be misleading. We consider two popular machine learning benchmark datasets for evaluation of heterogeneous treatment effect estimators. We highlight that the inherent characteristics of the benchmark datasets favor some algorithms over others.
  arXiv  Detail & Related papers  (2021-07-28T13:21:27Z)
• Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model [71.9860741092209]
  Clinical researchers often select among and evaluate risk prediction models. Standard metrics calculated from retrospective data are only related to model utility under certain assumptions. When predictions are delivered repeatedly throughout time, the relationship between standard metrics and utility is further complicated.
  arXiv  Detail & Related papers  (2020-06-02T16:26:49Z)
• Machine learning for causal inference: on the use of cross-fit estimators [77.34726150561087]
  Doubly-robust cross-fit estimators have been proposed to yield better statistical properties. We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE) When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
  arXiv  Detail & Related papers  (2020-04-21T23:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.