Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions

• URL: http://arxiv.org/abs/2607.25241v1
• Date: Tue, 28 Jul 2026 03:32:31 GMT
• Title: Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions
• Abstract summary: offline reinforcement learning algorithms typically assume that the actions in the dataset are observed without error.<n>In many real-world applications, the true actions are unobserved and only noisy proxies are available.<n>We study off-policy evaluation in infinite-horizon discounted decision processes with hidden actions.
• Score: 20.30528891217533
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard offline reinforcement learning (RL) algorithms typically assume that the actions in the dataset are observed without error. However, in many real-world applications, the true actions are unobserved and only noisy proxies are available, causing existing RL methods to yield biased and potentially misleading conclusions. We study off-policy evaluation in infinite-horizon discounted Markov decision processes with hidden actions. By leveraging the next-state variable as a natural proxy for the unobserved action, we establish identification of the policy value and propose an influence-function-based estimator called LURE (Learning from the Unseen: Robust Estimator). LURE is multiply robust, remaining consistent under several combinations of correctly specified nuisance components, and is asymptotically normal, enabling valid statistical inference. To our knowledge, this is the first work to address offline RL with hidden actions. We demonstrate LURE's effectiveness through simulations and a sepsis management application using the MIMIC-III database.

Related papers

• Causal Flow Q-Learning for Robust Offline Reinforcement Learning [53.63254824501714]
  We introduce a practical implementation that learns expressive flow-matching policies from confounded demonstrations.<n>Our proposed confounding-robust augmentation procedure achieves a success rate 120% that of confounding-unaware, state-of-the-art offline RL methods.
  arXiv  Detail & Related papers  (2026-02-02T21:50:52Z)
• Membership Inference Attacks from Causal Principles [24.370456956570873]
  We frame MIA evaluation as a causal inference problem, defining memorization as the causal effect of including a data point in the training set.<n>We propose practical estimators for multi-run, one-run, and zero-run regimes with non-asymptotic consistency guarantees.
  arXiv  Detail & Related papers  (2026-02-02T21:17:28Z)
• Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting [40.80967570661867]
  Adapting language models to new tasks via post-training carries the risk of degrading existing capabilities.<n>We compare the forgetting patterns of two widely adopted post-training methods: supervised fine-tuning (SFT) and reinforcement learning (RL)<n>RL leads to less forgetting than SFT while achieving comparable or higher target task performance.
  arXiv  Detail & Related papers  (2025-10-21T17:59:41Z)
• Reference-Specific Unlearning Metrics Can Hide the Truth: A Reality Check [60.77691669644931]
  We propose Functional Alignment for Distributional Equivalence (FADE), a novel metric that measures distributional similarity between unlearned and reference models.<n>We show that FADE captures functional alignment across the entire output distribution, providing a principled assessment of genuine unlearning.<n>These findings expose fundamental gaps in current evaluation practices and demonstrate that FADE provides a more robust foundation for developing and assessing truly effective unlearning methods.
  arXiv  Detail & Related papers  (2025-10-14T20:50:30Z)
• Outcome-Based Online Reinforcement Learning: Algorithms and Fundamental Limits [58.63897489864948]
  Reinforcement learning with outcome-based feedback faces a fundamental challenge.<n>How do we assign credit to the right actions?<n>This paper provides the first comprehensive analysis of this problem in online RL with general function approximation.
  arXiv  Detail & Related papers  (2025-05-26T17:44:08Z)
• LLM Unlearning via Neural Activation Redirection [24.157334866277534]
  We propose LUNAR, a novel unlearning method grounded in the Linear Representation Hypothesis.<n>We show that LUNAR achieves state-of-the-art unlearning performance and superior controllability.
  arXiv  Detail & Related papers  (2025-02-11T03:23:22Z)
• Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare [38.42691031505782]
  We propose a form of linear Q-function decomposition induced by factored action spaces. Our approach can help an agent make more accurate inferences within underexplored regions of the state-action space.
  arXiv  Detail & Related papers  (2023-05-02T19:13:10Z)
• Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning [96.72185761508668]
  Planning at Test-time (IMPLANT) is a new meta-algorithm for imitation learning. We demonstrate that IMPLANT significantly outperforms benchmark imitation learning approaches on standard control environments.
  arXiv  Detail & Related papers  (2022-04-07T17:16:52Z)
• Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning [30.337391523928396]
  We explore a restricted class of MDPs to obtain guarantees for offline reinforcement learning. We discuss algorithms that exploit the Action Impact Regularity (AIR) property, and provide a theoretical analysis for an algorithm based on Fitted-Q Iteration. We demonstrate that the algorithm outperforms existing offline reinforcement learning algorithms across different data collection policies in simulated and real world environments.
  arXiv  Detail & Related papers  (2021-11-15T20:14:18Z)
• False Correlation Reduction for Offline Reinforcement Learning [115.11954432080749]
  We propose falSe COrrelation REduction (SCORE) for offline RL, a practically effective and theoretically provable algorithm. We empirically show that SCORE achieves the SoTA performance with 3.1x acceleration on various tasks in a standard benchmark (D4RL)
  arXiv  Detail & Related papers  (2021-10-24T15:34:03Z)
• A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning [33.623558899286635]
  In deep reinforcement learning (RL) systems, abnormal states pose significant risks by potentially triggering unpredictable behaviors and unsafe actions. We propose a novel Mahalanobis distance-based (MD) anomaly detection framework, called textitMDX, for deep RL algorithms. MDX simultaneously addresses random, adversarial, and out-of-distribution (OOD) state outliers in both offline and online settings.
  arXiv  Detail & Related papers  (2021-09-21T00:09:03Z)
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning [63.53407136812255]
  Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. Existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from out-of-distribution (OOD) actions or states. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that detects OOD state-action pairs and down-weights their contribution in the training objectives accordingly.
  arXiv  Detail & Related papers  (2021-05-17T20:16:46Z)
• Strictly Batch Imitation Learning by Energy-based Distribution Matching [104.33286163090179]
  Consider learning a policy purely on the basis of demonstrated behavior -- that is, with no access to reinforcement signals, no knowledge of transition dynamics, and no further interaction with the environment. One solution is simply to retrofit existing algorithms for apprenticeship learning to work in the offline setting. But such an approach leans heavily on off-policy evaluation or offline model estimation, and can be indirect and inefficient. We argue that a good solution should be able to explicitly parameterize a policy, implicitly learn from rollout dynamics, and operate in an entirely offline fashion.
  arXiv  Detail & Related papers  (2020-06-25T03:27:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.