Offline RL for Adaptive Policy Retrieval in Prior Authorization

• URL: http://arxiv.org/abs/2604.05125v1
• Date: Mon, 06 Apr 2026 19:40:26 GMT
• Title: Offline RL for Adaptive Policy Retrieval in Prior Authorization
• Abstract summary: We train policies using Conservative Q-Learning (CQL), Implicit Q-Learning (IQL), and Direct Preference Optimization (DPO)<n>CQL achieves 92% decision accuracy (+30 percentage points over the best fixed-$K$ baseline) via exhaustive retrieval.<n>IQL matches the best baseline accuracy using 44% fewer retrieval steps and achieves the only positive episodic return among all policies.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prior authorization (PA) requires interpretation of complex and fragmented coverage policies, yet existing retrieval-augmented systems rely on static top-$K$ strategies with fixed numbers of retrieved sections. Such fixed retrieval can be inefficient and gather irrelevant or insufficient information. We model policy retrieval for PA as a sequential decision-making problem, formulating adaptive retrieval as a Markov Decision Process (MDP). In our system, an agent iteratively selects policy chunks from a top-$K$ candidate set or chooses to stop and issue a decision. The reward balances decision correctness against retrieval cost, capturing the trade-off between accuracy and efficiency. We train policies using Conservative Q-Learning (CQL), Implicit Q-Learning (IQL), and Direct Preference Optimization (DPO) in an offline RL setting on logged trajectories generated from baseline retrieval strategies over synthetic PA requests derived from publicly available CMS coverage data. On a corpus of 186 policy chunks spanning 10 CMS procedures, CQL achieves 92% decision accuracy (+30 percentage points over the best fixed-$K$ baseline) via exhaustive retrieval, while IQL matches the best baseline accuracy using 44% fewer retrieval steps and achieves the only positive episodic return among all policies. Transition-level DPO matches CQL's 92% accuracy while using 47% fewer retrieval steps (10.6 vs. 20.0), occupying a "selective-accurate" region on the Pareto frontier that dominates both CQL and BC. A behavioral cloning baseline matches CQL, confirming that advantage-weighted or preference-based policy extraction is needed to learn selective retrieval. Lambda ablation over step costs $λ\in \{0.05, 0.1, 0.2\}$ reveals a clear accuracy-efficiency inflection: only at $λ= 0.2$ does CQL transition from exhaustive to selective retrieval.

Related papers

• Peng's Q($λ$) for Conservative Value Estimation in Offline Reinforcement Learning [35.18584220158021]
  We propose a model-free offline multi-step reinforcement learning algorithm, Conservative Peng's Q($$) (CPQL)<n>Our algorithm adapts the Peng's Q($$) (PQL) operator for conservative value estimation as an alternative to the Bellman operator.<n>CPQL simultaneously mitigates over-pessimistic value estimation, achieves performance greater than (or equal to) that of the behavior policy, and provides near-optimal performance guarantees.
  arXiv  Detail & Related papers  (2026-05-14T12:48:44Z)
• Learning Partial Action Replacement in Offline MARL [11.861550409939818]
  Partial Action Replacement mitigates this by anchoring a subset of agents to dataset actions.<n>We introduce PLCQL, a framework that formulates PAR subset selection as a contextual bandit problem.<n>We prove a value-error bound showing that the estimation error scales linearly with the expected number of deviating agents.
  arXiv  Detail & Related papers  (2026-03-30T15:28:13Z)
• Efficient Context Selection for Long-Context QA: No Tuning, No Iteration, Just Adaptive-$k$ [13.173467492641485]
  We present Adaptive-$k$, a simple and effective single-pass method that adaptively selects the number of passages based on the similarity scores between the query and the candidate passages.<n>It does not require model fine-tuning, extra LLM inferences or changes to existing retriever-reader pipelines.<n>It improves accuracy across five LCLMs and two embedding models, highlighting that dynamically adjusting context size leads to more efficient and accurate QA.
  arXiv  Detail & Related papers  (2025-06-10T06:11:01Z)
• Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs [54.05511925104712]
  We propose a simple, effective, and data-efficient method called Step-DPO. Step-DPO treats individual reasoning steps as units for preference optimization rather than evaluating answers holistically. Our findings demonstrate that as few as 10K preference data pairs and fewer than 500 Step-DPO training steps can yield a nearly 3% gain in accuracy on MATH for models with over 70B parameters.
  arXiv  Detail & Related papers  (2024-06-26T17:43:06Z)
• AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization [17.795383012026118]
  Implicit Q-learning serves as a strong baseline for offline RL.<n>We introduce a different way to solve the implicit policy-finding problem (IPF) by formulating this problem as an optimization problem.<n>Compared with IQL and IDQL, we find our method keeps the simplicity of IQL and solves the implicit policy-finding problem.
  arXiv  Detail & Related papers  (2024-05-28T14:01:03Z)
• Query Performance Prediction using Relevance Judgments Generated by Large Language Models [53.97064615557883]
  We propose a new Query performance prediction (QPP) framework using automatically generated relevance judgments (QPP-GenRE)<n>QPP-GenRE decomposes QPP into independent subtasks of predicting relevance of each item in a ranked list to a given query.<n>We predict an item's relevance by using open-source large language models (LLMs) to ensure scientific relevance.
  arXiv  Detail & Related papers  (2024-04-01T09:33:05Z)
• IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies [72.4573167739712]
  Implicit Q-learning (IQL) trains a Q-function using only dataset actions through a modified Bellman backup. It is unclear which policy actually attains the values represented by this trained Q-function. We introduce Implicit Q-learning (IDQL), combining our general IQL critic with the policy extraction method.
  arXiv  Detail & Related papers  (2023-04-20T18:04:09Z)
• Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization [90.9780151608281]
  In-sample learning (IQL) improves the policy by quantile regression using only data samples. We make a key finding that the in-sample learning paradigm arises under the textitImplicit Value Regularization (IVR) framework. We propose two practical algorithms, Sparse $Q$-learning (EQL) and Exponential $Q$-learning (EQL), which adopt the same value regularization used in existing works.
  arXiv  Detail & Related papers  (2023-03-28T08:30:01Z)
• SEAM: Searching Transferable Mixed-Precision Quantization Policy through Large Margin Regularization [50.04951511146338]
  Mixed-precision quantization (MPQ) suffers from the time-consuming process of searching the optimal bit-width allocation for each layer. This paper proposes a novel method for efficiently searching for effective MPQ policies using a small proxy dataset.
  arXiv  Detail & Related papers  (2023-02-14T05:47:45Z)
• Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient [62.24615324523435]
  This paper provides a statistical analysis of high-dimensional batch Reinforcement Learning (RL) using sparse linear function approximation. When there is a large number of candidate features, our result sheds light on the fact that sparsity-aware methods can make batch RL more sample efficient.
  arXiv  Detail & Related papers  (2020-11-08T16:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.