Comment: Entropy Learning for Dynamic Treatment Regimes

• URL: http://arxiv.org/abs/2004.02778v1
• Date: Mon, 6 Apr 2020 16:11:05 GMT
• Title: Comment: Entropy Learning for Dynamic Treatment Regimes
• Abstract summary: JSLZ's approach leverages a rejection-and-sampling estimate of the value of a given decision rule based on inverse probability (IPW) and its interpretation as a weighted (or cost-sensitive) classification. Their use of smooth classification surrogates enables their careful approach to analyzing distributions. The IPW estimate is problematic as it leads to weights that discard most of the data and are extremely variable on whatever remains.
• Score: 58.442274475425144
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: I congratulate Profs. Binyan Jiang, Rui Song, Jialiang Li, and Donglin Zeng (JSLZ) for an exciting development in conducting inferences on optimal dynamic treatment regimes (DTRs) learned via empirical risk minimization using the entropy loss as a surrogate. JSLZ's approach leverages a rejection-and-importance-sampling estimate of the value of a given decision rule based on inverse probability weighting (IPW) and its interpretation as a weighted (or cost-sensitive) classification. Their use of smooth classification surrogates enables their careful approach to analyzing asymptotic distributions. However, even for evaluation purposes, the IPW estimate is problematic as it leads to weights that discard most of the data and are extremely variable on whatever remains. In this comment, I discuss an optimization-based alternative to evaluating DTRs, review several connections, and suggest directions forward. This extends the balanced policy evaluation approach of Kallus (2018a) to the longitudinal setting.

Related papers

• IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning [57.16513298507272]
  We propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR)<n>IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization.<n>For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional.
  arXiv  Detail & Related papers  (2026-08-11T08:20:02Z)
• Non-asymptotic Convergence of Stochastic Gradient Descent in Score-based Generative Models [5.501082429793833]
  Score-based Generative Models (SGMs) have achieved impressive performance in data generation across a wide range of applications.<n>These models are typically trained by minimizing a denoising objective, yet optimization guarantees with gradients remain limited.
  arXiv  Detail & Related papers  (2026-07-06T08:07:15Z)
• LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models [34.349722314481824]
  Group Relative Policy Optimization is prized for its efficacy in foregoing an explicit value-critic.<n>The method's reliance on a monolithic statistical baseline, such as the group mean, collapses the relational topology of the trajectory space into a single scalar.<n>We introduce a novel framework, Lambda Policy Optimization (LambdaPO), that addresses this information-theoretic bottleneck.
  arXiv  Detail & Related papers  (2026-05-19T06:10:24Z)
• Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settings [13.1234934131475]
  We propose a policy-aware re parameterization of Iterative Conditional Expectation (ICE) Q-functions that enables joint estimation through shared representations.<n>We implement this approach in the Policy-Encoded Q Network (PEQ-Net), an architecture centered on a shared policy encoder.<n> Experiments on semi-synthetic datasets demonstrate that PEQ-Net consistently outperforms existing ICE-based methods, achieving substantial reductions in root-mean-square error.
  arXiv  Detail & Related papers  (2026-05-14T02:33:58Z)
• EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training [69.32453275232662]
  We show that a learned critic can inject estimation noise that exceeds the state signal it captures, increasing rather than reducing advantage variance.<n>We propose Explained Variance Policy Optimization (EVPO), which monitors batch-level EV at each training step and adaptively switches between critic-based and batch-mean advantage estimation.
  arXiv  Detail & Related papers  (2026-04-21T14:07:39Z)
• Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Sampling [40.94400211806987]
  We propose a policy-driven ZO framework that treats the sampling distribution over perturbation directions as a learnable policy.<n>We show that learned sampling improves quality gradient information and relax the explicit dependence on $d$ in convergence bounds.<n>Our results suggest that adaptive direction sampling is a promising route to make ZO fine-tuning viable at scale.
  arXiv  Detail & Related papers  (2026-02-14T08:01:41Z)
• AVG-DICE: Stationary Distribution Correction by Regression [7.193870502672509]
  Off-policy policy evaluation (OPE) has long suffered from stationary state distribution mismatch. We introduce AVG-DICE, a computationally simple Monte Carlo estimator for the density ratio. In our experiments, AVG-DICE is at least as accurate as state-of-the-art estimators and sometimes offers orders-of-magnitude improvements.
  arXiv  Detail & Related papers  (2025-03-03T23:14:02Z)
• Uncertainty-Penalized Direct Preference Optimization [52.387088396044206]
  We develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. We show improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
  arXiv  Detail & Related papers  (2024-10-26T14:24:37Z)
• Truncating Trajectories in Monte Carlo Policy Evaluation: an Adaptive Approach [51.76826149868971]
  Policy evaluation via Monte Carlo simulation is at the core of many MC Reinforcement Learning (RL) algorithms. We propose as a quality index a surrogate of the mean squared error of a return estimator that uses trajectories of different lengths. We present an adaptive algorithm called Robust and Iterative Data collection strategy Optimization (RIDO)
  arXiv  Detail & Related papers  (2024-10-17T11:47:56Z)
• Leveraging Nested MLMC for Sequential Neural Posterior Estimation with Intractable Likelihoods [0.7373617024876725]
  SNPE techniques are used to learn the posterior from proposed simulations using neural network-based conditional density estimators.<n>As a SNPE technique, the automatic nested posterior transformation (APT) method performs notably and scales to high dimensional data.<n>In this paper, we propose a new APT method to estimate the involved expectation instead.
  arXiv  Detail & Related papers  (2024-01-30T06:29:41Z)
• On Training Implicit Meta-Learning With Applications to Inductive Weighing in Consistency Regularization [0.0]
  Implicit meta-learning (IML) requires computing $2nd$ order gradients, particularly the Hessian. Various approximations for the Hessian were proposed but a systematic comparison of their compute cost, stability, generalization of solution found and estimation accuracy were largely overlooked. We show how training a "Confidence Network" to extract domain specific features can learn to up-weigh useful images and down-weigh out-of-distribution samples.
  arXiv  Detail & Related papers  (2023-10-28T15:50:03Z)
• A Semiparametric Instrumented Difference-in-Differences Approach to Policy Learning [2.1989182578668243]
  We propose a general instrumented difference-in-differences (DiD) approach for learning the optimal treatment policy. Specifically, we establish identification results using a binary instrumental variable (IV) when the parallel trends assumption fails to hold. We also construct a Wald estimator, novel inverse probability estimators, and a class of semi efficient and multiply robust estimators.
  arXiv  Detail & Related papers  (2023-10-14T09:38:32Z)
• Post Reinforcement Learning Inference [22.117487428829488]
  We consider estimation and inference using data collected from reinforcement learning algorithms. We propose a weighted Z-estimation approach with carefully designed adaptive weights to stabilize the time-varying variance. Primary applications include dynamic treatment effect estimation and dynamic off-policy evaluation.
  arXiv  Detail & Related papers  (2023-02-17T12:53:15Z)
• Revisiting Estimation Bias in Policy Gradients for Deep Reinforcement Learning [0.0]
  We revisit the estimation bias in policy gradients for the discounted episodic Markov decision process (MDP) from Deep Reinforcement Learning perspective. One of the major policy biases is the state distribution shift. We show that, despite such state distribution shift, the policy gradient estimation bias can be reduced in the following three ways.
  arXiv  Detail & Related papers  (2023-01-20T06:46:43Z)
• An Investigation of the Bias-Variance Tradeoff in Meta-Gradients [53.28925387487846]
  Hessian estimation always adds bias and can also add variance to meta-gradient estimation. We study the bias and variance tradeoff arising from truncated backpropagation and sampling correction.
  arXiv  Detail & Related papers  (2022-09-22T20:33:05Z)
• Variance-Aware Off-Policy Evaluation with Linear Function Approximation [85.75516599931632]
  We study the off-policy evaluation problem in reinforcement learning with linear function approximation. We propose an algorithm, VA-OPE, which uses the estimated variance of the value function to reweight the Bellman residual in Fitted Q-Iteration.
  arXiv  Detail & Related papers  (2021-06-22T17:58:46Z)
• On Signal-to-Noise Ratio Issues in Variational Inference for Deep Gaussian Processes [55.62520135103578]
  We show that the gradient estimates used in training Deep Gaussian Processes (DGPs) with importance-weighted variational inference are susceptible to signal-to-noise ratio (SNR) issues. We show that our fix can lead to consistent improvements in the predictive performance of DGP models.
  arXiv  Detail & Related papers  (2020-11-01T14:38:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.