Related papers: Assessing model error in counterfactual worlds

Assessing model error in counterfactual worlds

URL: http://arxiv.org/abs/2512.00836v1
Date: Sun, 30 Nov 2025 11:08:43 GMT
Title: Assessing model error in counterfactual worlds
Abstract summary: We argue that differences between projections and observations come from two sources: scenario deviation and model miscalibration.<n>Here we present and contrast three approaches for estimating this error, and demonstrate the benefits and limitations of each in a simulation experiment.<n>We provide recommendations for the estimation of counterfactual error and discuss the components of scenario design that are required to make scenario projections evaluable.
Score: 0.0
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Counterfactual scenario modeling exercises that ask "what would happen if?" are one of the most common ways we plan for the future. Despite their ubiquity in planning and decision making, scenario projections are rarely evaluated retrospectively. Differences between projections and observations come from two sources: scenario deviation and model miscalibration. We argue the latter is most important for assessing the value of models in decision making, but requires estimating model error in counterfactual worlds. Here we present and contrast three approaches for estimating this error, and demonstrate the benefits and limitations of each in a simulation experiment. We provide recommendations for the estimation of counterfactual error and discuss the components of scenario design that are required to make scenario projections evaluable.

Related papers

From Black-box to Causal-box: Towards Building More Interpretable Models [57.23201263629627]
We introduce the notion of causal interpretability, which formalizes when counterfactual queries can be evaluated from a specific class of models.<n>We derive a complete graphical criterion that determines whether a given model architecture supports a given counterfactual query.
arXiv Detail & Related papers (2025-10-24T20:03:18Z)
A Probabilistic Perspective on Unlearning and Alignment for Large Language Models [48.96686419141881]
We introduce the first formal probabilistic evaluation framework for Large Language Models (LLMs)<n> Namely, we propose novel metrics with high probability guarantees concerning the output distribution of a model.<n>Our metrics are application-independent and allow practitioners to make more reliable estimates about model capabilities before deployment.
arXiv Detail & Related papers (2024-10-04T15:44:23Z)
Bounding-Box Inference for Error-Aware Model-Based Reinforcement Learning [4.185571779339683]
In model-based reinforcement learning, simulated experiences are often treated as equivalent to experience from the real environment. We show that best results require distribution insensitive inference to estimate the uncertainty over model-based updates. We find that bounding-box inference can reliably support effective selective planning.
arXiv Detail & Related papers (2024-06-23T04:23:15Z)
Scenario-based Evaluation of Prediction Models for Automated Vehicles [0.0]
We argue that following evaluation practices in safety assessment for automated vehicles should be performed in a scenario-based fashion. We categorize trajectories of Open Motion dataset according to the type of movement they capture. Results show that common evaluation methods are insufficient and the assessment should be performed depending on the application in which the model will operate.
arXiv Detail & Related papers (2022-10-11T09:45:43Z)
Probabilistic Modeling for Human Mesh Recovery [73.11532990173441]
This paper focuses on the problem of 3D human reconstruction from 2D evidence. We recast the problem as learning a mapping from the input to a distribution of plausible 3D poses.
arXiv Detail & Related papers (2021-08-26T17:55:11Z)
Planning with Expectation Models for Control [8.238241895275232]
Wan et al. considered only planning for prediction to evaluate a fixed policy. We prove that planning with an expectation model must update a state-value function, not an action-value function.
arXiv Detail & Related papers (2021-04-17T13:37:14Z)
The Causal Learning of Retail Delinquency [13.866975878174207]
This paper focuses on the expected difference in borrower's repayment when there is a change in the lender's credit decisions. We propose another approach to construct the estimators such that the error can be greatly reduced. We find that the relative reduction of estimation error is strikingly substantial if the causal effects are accounted for correctly.
arXiv Detail & Related papers (2020-12-17T08:46:01Z)
Forethought and Hindsight in Credit Assignment [62.05690959741223]
We work to understand the gains and peculiarities of planning employed as forethought via forward models or as hindsight operating with backward models. We investigate the best use of models in planning, primarily focusing on the selection of states in which predictions should be (re)-evaluated.
arXiv Detail & Related papers (2020-10-26T16:00:47Z)
Counterfactual Predictions under Runtime Confounding [74.90756694584839]
We study the counterfactual prediction task in the setting where all relevant factors are captured in the historical data. We propose a doubly-robust procedure for learning counterfactual prediction models in this setting.
arXiv Detail & Related papers (2020-06-30T15:49:05Z)
Decision-Making with Auto-Encoding Variational Bayes [71.44735417472043]
We show that a posterior approximation distinct from the variational distribution should be used for making decisions. Motivated by these theoretical results, we propose learning several approximate proposals for the best model. In addition to toy examples, we present a full-fledged case study of single-cell RNA sequencing.
arXiv Detail & Related papers (2020-02-17T19:23:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.