Quotient DAGs for Off-Policy Evaluation:Forward-Flow Importance Sampling and Exact Slate Propensities

• URL: http://arxiv.org/abs/2605.29500v1
• Date: Thu, 28 May 2026 07:23:40 GMT
• Title: Quotient DAGs for Off-Policy Evaluation:Forward-Flow Importance Sampling and Exact Slate Propensities
• Abstract summary: Off-policy evaluation estimates how a target policy would perform using data collected by a different behavior policy.<n>Standard importance sampling reweights each logged trajectory, but it can treat details of the generation process as meaningful even when the evaluation target ignores them.<n>We introduce a quotient-DAG view that merges histories equivalent for evaluation and assigns weights using target-to-behavior forward-flow ratios.
• Score: 4.751925814658969
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Off-policy evaluation estimates how a target policy would perform using data collected by a different behavior policy, which is crucial when online testing is costly or risky, such as in recommendation or healthcare. Standard importance sampling reweights each logged trajectory, but it can treat details of the generation process as meaningful even when the evaluation target ignores them: for example, an autoregressive slate recommender may generate an ordered sequence of items while the reward and downstream estimator depend only on the unordered slate. This creates nuisance variance and a computational gap, since exact unordered slate propensities require summing over all generation orders. We introduce a quotient-DAG view that merges histories equivalent for evaluation and assigns weights using target-to-behavior forward-flow ratios on the merged graph. For slate recommendation under a set-sufficient next-item interface, this yields Forward-DP, a subset-DAG dynamic program that computes exact unordered propensities without factorial enumeration. The resulting propensity primitive enables practical propensity-based evaluation and model selection for context-dependent autoregressive slate loggers.

Related papers

• Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination [8.221134173409895]
  We propose textbfSA-AP (textbfStratified textbfAggregate of textbfPer-question textbfProbability textbfGaps): estimate each question's solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model's solve probability.
  arXiv  Detail & Related papers  (2026-08-07T15:37:03Z)
• OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation [38.52509364819271]
  OrderGrad is a family of likelihood-based gradient estimators for order-statistic objectives.<n>It recovers objectives such as VaR, CVaR, trimmed means, medians, and top-m/best-of-K criteria by changing only the rank weights.<n>We study the resulting estimator's variance behavior and evaluate it on tasks where mean optimization is mismatched to the deployment objective.
  arXiv  Detail & Related papers  (2026-06-04T12:34:15Z)
• Tailoring Strictly Proper Scoring Rules for Downstream Tasks: An Application to Causal Inference [48.78219918881965]
  We propose a principled framework for deriving task-specific strictly proper scoring rules by matching the local curvature of the downstream error metric.<n>We apply this to the Average Treatment Effect (ATE) estimation, deriving a closed-form loss and its corresponding canonical probability mapping.
  arXiv  Detail & Related papers  (2026-06-02T08:41:25Z)
• RAGR: Review-Augmented Generative Recommendation [61.29879991713178]
  Review-Augmented Generative Recommendation (RAGR) is a novel GR framework that incorporates review feedback directly into the generative user sequence.<n>RAGR yields consistent and significant gains over strong GR backbones across all metrics.
  arXiv  Detail & Related papers  (2026-05-17T05:21:23Z)
• Relative Score Policy Optimization for Diffusion Language Models [29.344961499429257]
  Diffusion large language models (dLLMs) offer a promising route to parallel and efficient text generation.<n>The lack of tractable sequence-level log-ratios forces existing methods to rely on high-variance ELBO-based approximations.<n>We propose textbfRelative textbfScore textbfPolicy textbfOptimization (RSPO) to calibrate noisy likelihood estimates in dLLMs.
  arXiv  Detail & Related papers  (2026-05-11T08:58:40Z)
• What If We Allocate Test-Time Compute Adaptively? [2.1713977971908944]
  Test-time scaling allocates inference computation uniformly, uses fixed sampling strategies, and applies verification only for reranking.<n>We propose a verifier-guided adaptive framework treating reasoning as iterative trajectory generation and selection.<n>Across datasets, our dynamic, PRM-guided approach consistently outperforms direct test-time scaling.
  arXiv  Detail & Related papers  (2026-02-01T07:30:22Z)
• Adaptive Sample Sharing for Linear Regression [1.8898307337832196]
  We study sample sharing in the case of ridge regression.<n>We introduce a principled, data-driven rule that decides how many samples from an auxiliary dataset to add to the target training set.<n>We validate the approach in synthetic and real datasets, observing consistent gains over strong baselines and single-task training.
  arXiv  Detail & Related papers  (2025-10-19T20:03:48Z)
• Aligning the Evaluation of Probabilistic Predictions with Downstream Value [2.6636053598505307]
  Metrics based solely on predictive performance often diverge from measures of real-world downstream impact.<n>We propose a data-driven method to learn a proxy evaluation function aligned with the downstream evaluation.<n>Our approach leverages weighted scoring rules parametrized by a neural network, where weighting is learned to align with the performance in the downstream task.
  arXiv  Detail & Related papers  (2025-08-25T17:41:27Z)
• Preference Trajectory Modeling via Flow Matching for Sequential Recommendation [50.077447974294586]
  Sequential recommendation predicts each user's next item based on their historical interaction sequence.<n>FlowRec is a simple yet effective sequential recommendation framework.<n>We construct a personalized behavior-based prior distribution to replace Gaussian noise and learn a vector field to model user preference trajectories.
  arXiv  Detail & Related papers  (2025-08-25T02:55:42Z)
• Time-series Generation by Contrastive Imitation [87.51882102248395]
  We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
  arXiv  Detail & Related papers  (2023-11-02T16:45:25Z)
• Post Reinforcement Learning Inference [20.521169740409263]
  We study estimation and inference using data collected by reinforcement learning algorithms.<n>We propose a weighted generalized method of moments (GMM) that uses adaptive weights to stabilize this variance.<n>Key applications include dynamic treatment effect estimation and dynamic off-policy evaluation.
  arXiv  Detail & Related papers  (2023-02-17T12:53:15Z)
• Sequential Recommendation via Stochastic Self-Attention [68.52192964559829]
  Transformer-based approaches embed items as vectors and use dot-product self-attention to measure the relationship between items. We propose a novel textbfSTOchastic textbfSelf-textbfAttention(STOSA) to overcome these issues. We devise a novel Wasserstein Self-Attention module to characterize item-item position-wise relationships in sequences.
  arXiv  Detail & Related papers  (2022-01-16T12:38:45Z)
• Off-Policy Evaluation via Adaptive Weighting with Data from Contextual Bandits [5.144809478361604]
  We improve the doubly robust (DR) estimator by adaptively weighting observations to control its variance. We provide empirical evidence for our estimator's improved accuracy and inferential properties relative to existing alternatives.
  arXiv  Detail & Related papers  (2021-06-03T17:54:44Z)
• Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient [62.24615324523435]
  This paper provides a statistical analysis of high-dimensional batch Reinforcement Learning (RL) using sparse linear function approximation. When there is a large number of candidate features, our result sheds light on the fact that sparsity-aware methods can make batch RL more sample efficient.
  arXiv  Detail & Related papers  (2020-11-08T16:48:02Z)
• Comment: Entropy Learning for Dynamic Treatment Regimes [58.442274475425144]
  JSLZ's approach leverages a rejection-and-sampling estimate of the value of a given decision rule based on inverse probability (IPW) and its interpretation as a weighted (or cost-sensitive) classification. Their use of smooth classification surrogates enables their careful approach to analyzing distributions. The IPW estimate is problematic as it leads to weights that discard most of the data and are extremely variable on whatever remains.
  arXiv  Detail & Related papers  (2020-04-06T16:11:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.