Counterfactual Evaluation of Temporal Observation Protocols

• URL: http://arxiv.org/abs/2608.22221v1
• Date: Sun, 23 Aug 2026 05:05:16 GMT
• Title: Counterfactual Evaluation of Temporal Observation Protocols
• Abstract summary: We study whether data collected under a realised observation protocol determine the predictive value of alternatives that were never deployed.<n>We show that even infinite benchmark data need not determine this value.<n>We develop a value-specific identification theory in which only latent ambiguity that changes the alternative's value matters.
• Score: 1.2292186162350942
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study counterfactual protocol evaluation: whether data collected under a realised observation protocol determine the predictive value of alternatives that were never deployed. Protocol value is the population $R^2$ of the Bayes-optimal predictor of a fixed trajectory-level target from the measurements an alternative would collect. We show that even infinite benchmark data need not determine this value: distinct latent covariance structures can induce the same benchmark measurement--target law while assigning different values to the same alternative. We develop a value-specific identification theory in which only latent ambiguity that changes the alternative's value matters. For linear targets, invisible covariance directions certify non-identification, while targeted measurements can restore identification without recovering the full latent covariance; an exact permutation construction extends the result to nonlinear aggregate targets. With finite dense calibration data, uniform error bounds control protocol-selection regret and distinguishable value gaps. Exact marginal gains then support cost-constrained, target-aware observation design. Simulations and retrospective analyses of Sleep-EDF and Long-Term AF show that broad temporal-layout differences can be more reliably distinguished than fine placements selected from finite data. Together, these results connect identification, calibration resolution and observation design for undeployed protocols.

Related papers

• Marginal Coordinate Test for Fréchet Regression with Random Objects [15.929485404318557]
  We develop a test for regression with Euclidean predictors and a random-object response in a separable metric space.<n>We establish a weighted centered chi-square null limit, wild bootstrap validity, and consistency against fixed detectable alternatives.<n>For simultaneous inference, truncated p-to-e calibration combined with e-BH provides false discovery rate control under general dependence.
  arXiv  Detail & Related papers  (2026-08-31T11:48:12Z)
• Why Does the Future Branch? Identifiable Closure Tests for Stochastic Physical World Models [0.0]
  ClosurePairs crosses compatible microstates with repeated disturbances.<n>It recovers source attribution at unchanged likelihood.<n>It reduces equal-budget decomposition error in a nonlinear interaction benchmark.
  arXiv  Detail & Related papers  (2026-08-01T11:07:24Z)
• Unified Precision-Guaranteed Stopping Rules for Contextual Learning [8.604741134620559]
  Contextual learning seeks to learn a decision policy that maps an individual's characteristics to an action through data collection.<n>We study this question under two precision criteria: a context-wise criterion and an aggregate policy-value criterion.<n>We develop unified stopping rules for contextual learning with unknown sampling variances in both unstructured and structured linear settings.
  arXiv  Detail & Related papers  (2026-04-09T07:30:15Z)
• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• FIVA: Federated Inverse Variance Averaging for Universal CT Segmentation with Uncertainty Estimation [4.544160712377809]
  This work presents a novel federated learning approach to achieve universal segmentation across diverse abdominal CT datasets.<n>The proposed method quantifies prediction uncertainty by propagating the uncertainty from the model weights.<n> Experimental evaluations demonstrate the effectiveness of this approach in improving the quality of federated aggregation and uncertainty-weighted inference.
  arXiv  Detail & Related papers  (2025-08-08T11:34:01Z)
• Error-quantified Conformal Inference for Time Series [55.11926160774831]
  Uncertainty quantification in time series prediction is challenging due to the temporal dependence and distribution shift on sequential data.<n>We propose itError-quantified Conformal Inference (ECI) by smoothing the quantile loss function.<n>ECI can achieve valid miscoverage control and output tighter prediction sets than other baselines.
  arXiv  Detail & Related papers  (2025-02-02T15:02:36Z)
• Distributional Shift-Aware Off-Policy Interval Estimation: A Unified Error Quantification Framework [8.572441599469597]
  We study high-confidence off-policy evaluation in the context of infinite-horizon Markov decision processes. The objective is to establish a confidence interval (CI) for the target policy value using only offline data pre-collected from unknown behavior policies. We show that our algorithm is sample-efficient, error-robust, and provably convergent even in non-linear function approximation settings.
  arXiv  Detail & Related papers  (2023-09-23T06:35:44Z)
• Adapting to Latent Subgroup Shifts via Concepts and Proxies [82.01141290360562]
  We show that the optimal target predictor can be non-parametrically identified with the help of concept and proxy variables available only in the source domain. For continuous observations, we propose a latent variable model specific to the data generation process at hand.
  arXiv  Detail & Related papers  (2022-12-21T18:30:22Z)
• Meta Learning Low Rank Covariance Factors for Energy-Based Deterministic Uncertainty [58.144520501201995]
  Bi-Lipschitz regularization of neural network layers preserve relative distances between data instances in the feature spaces of each layer. With the use of an attentive set encoder, we propose to meta learn either diagonal or diagonal plus low-rank factors to efficiently construct task specific covariance matrices. We also propose an inference procedure which utilizes scaled energy to achieve a final predictive distribution.
  arXiv  Detail & Related papers  (2021-10-12T22:04:19Z)
• CertainNet: Sampling-free Uncertainty Estimation for Object Detection [65.28989536741658]
  Estimating the uncertainty of a neural network plays a fundamental role in safety-critical settings. In this work, we propose a novel sampling-free uncertainty estimation method for object detection. We call it CertainNet, and it is the first to provide separate uncertainties for each output signal: objectness, class, location and size.
  arXiv  Detail & Related papers  (2021-10-04T17:59:31Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.