ATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialogue

• URL: http://arxiv.org/abs/2603.02216v1
• Date: Tue, 10 Feb 2026 07:47:23 GMT
• Title: ATPO: Adaptive Tree Policy Optimization for Multi-Turn Medical Dialogue
• Abstract summary: We propose a novel uncertainty-aware Adaptive Tree Policy Optimization (ATPO) algorithm.<n>Our method adaptively allocates the rollout budget to states with high uncertainty, quantified by a composite metric of Bellman error and action-value variance.<n>Experiments on three public medical dialogue benchmarks demonstrate that our algorithm significantly outperforms several strong baselines.
• Score: 9.221285189821396
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Effective information seeking in multi-turn medical dialogues is critical for accurate diagnosis, especially when dealing with incomplete information. Aligning Large Language Models (LLMs) for these interactive scenarios is challenging due to the uncertainty inherent in user-agent interactions, which we formulate as a Hierarchical Markov Decision Process (H-MDP). While conventional Reinforcement Learning (RL) methods like Group Relative Policy Optimization (GRPO) struggle with long-horizon credit assignment and Proximal Policy Optimization (PPO) suffers from unstable value estimation in this context, we propose a novel uncertainty-aware Adaptive Tree Policy Optimization (ATPO) algorithm. Our method adaptively allocates the rollout budget to states with high uncertainty, quantified by a composite metric of Bellman error and action-value variance. This strategy enables more accurate value estimation, while fostering more efficient and diverse exploration. To mitigate the high computational cost of tree-based RL, we introduce two key optimizations: an uncertainty-guided pruning mechanism to minimize the number of rollouts, and an asynchronous search architecture that leverages KV cache reuse to maximize inference throughput. Extensive experiments on three public medical dialogue benchmarks demonstrate that our algorithm significantly outperforms several strong baselines, culminating in Qwen3-8B model surpassing the much larger GPT-4o ($+0.92\%$ accuracy).

Related papers

• Optimizing Sparse Outcomes Through Dense Behavioral Signals via Value-Guided Preference Distillation [5.277024349608834]
  We formulate long-horizon dialogue optimization as a multi-objective reinforcement learning problem.<n>We show that a scalarized composite of dense auxiliary behavioral signals enables effective credit assignment and optimization of sparse outcomes.<n>We demonstrate that our distilled policy significantly improves long-term user retention, while simultaneously enhancing the positive behaviors and therapeutic-process markers.
  arXiv  Detail & Related papers  (2026-09-13T16:32:54Z)
• SAGE: Stochastic Prompt Optimization via Agent-Guided Exploration [4.5848302154106815]
  We introduce SPO (Stochastic Prompt Optimization), a framework for search over prompt space.<n>We compare three strategies of increasing sophistication: error-informed random search, a genetic algorithm with evolutionary operators, and SAGE (SPO via Agent-Guided Exploration)<n>We argue that coupling qualitative diagnosis with quantitative validation is what makes agentic optimization effective for open-ended task-oriented dialogue.
  arXiv  Detail & Related papers  (2026-06-17T10:25:25Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference [60.958331943869126]
  ODAR-Expert is an adaptive routing framework that optimize the accuracy-efficiency trade-off via principled resource allocation.<n>We show strong and consistent gains, including 98.2% accuracy on MATH and 54.8% on Humanity's Last Exam.
  arXiv  Detail & Related papers  (2026-02-27T05:22:01Z)
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization [133.27496265096445]
  We show how to apply Group Relative Policy Optimization under multi-reward setting without examining its suitability.<n>We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues.<n>GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
  arXiv  Detail & Related papers  (2026-01-08T18:59:24Z)
• SPOT: Scalable Policy Optimization with Trees for Markov Decision Processes [3.1382171194541306]
  Interpretable reinforcement learning policies are essential for high-stakes decision-making.<n>We propose SPOT, a novel method for computing decision tree policies.<n>We employ a reduced-space branch-and-bound approach that decouples the MDP dynamics from tree-structure constraints.
  arXiv  Detail & Related papers  (2025-10-22T04:57:23Z)
• Hierarchical Budget Policy Optimization for Adaptive Reasoning [49.621779447691665]
  We present Hierarchical Budget Policy Optimization (HBPO), a reinforcement learning framework that enables models to learn problem-specific reasoning depths without sacrificing capability.<n>HBPO partitions the exploration space into budget-constrained hierarchies (512-2560 tokens), each with differentiated reward structures that preserve both efficiency incentives and reasoning capabilities.<n>Extensive experiments demonstrate that HBPO reduces average token usage by up to 60.6% while improving accuracy by 3.14% across four reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-07-21T17:52:34Z)
• Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Bounds [20.97190146319937]
  In brachytherapy, clinicians must balance maximizing tumor coverage against strict organ dose limits.<n>Current methods often lack systematic approaches to iteratively refine these multi-faceted preference structures.<n>We present Active-MoSH, an interactive local-global framework designed for this process.
  arXiv  Detail & Related papers  (2025-06-27T03:44:20Z)
• Q-VLM: Post-training Quantization for Large Vision-Language Models [73.19871905102545]
  We propose a post-training quantization framework of large vision-language models (LVLMs) for efficient multi-modal inference.<n>We mine the cross-layer dependency that significantly influences discretization errors of the entire vision-language model, and embed this dependency into optimal quantization strategy.<n> Experimental results demonstrate that our method compresses the memory by 2.78x and increase generate speed by 1.44x about 13B LLaVA model without performance degradation.
  arXiv  Detail & Related papers  (2024-10-10T17:02:48Z)
• Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization [75.1240295759264]
  We propose an effective framework for Bridging and Modeling Correlations in pairwise data, named BMC.<n>We increase the consistency and informativeness of the pairwise preference signals through targeted modifications.<n>We identify that DPO alone is insufficient to model these correlations and capture nuanced variations.
  arXiv  Detail & Related papers  (2024-08-14T11:29:47Z)
• The Hitchhiker's Guide to Human Alignment with *PO [43.4130314879284]
  We focus on identifying the algorithm that, while being performant, is simultaneously more robust to varying hyper parameters. Our analysis reveals that the widely adopted DPO method consistently produces lengthy responses of inferior quality. Motivated by these findings, we propose an embarrassingly simple extension to the DPO algorithm, LN-DPO, resulting in more concise responses without sacrificing quality.
  arXiv  Detail & Related papers  (2024-07-21T17:35:20Z)
• Provably Efficient UCB-type Algorithms For Learning Predictive State Representations [55.00359893021461]
  The sequential decision-making problem is statistically learnable if it admits a low-rank structure modeled by predictive state representations (PSRs) This paper proposes the first known UCB-type approach for PSRs, featuring a novel bonus term that upper bounds the total variation distance between the estimated and true models. In contrast to existing approaches for PSRs, our UCB-type algorithms enjoy computational tractability, last-iterate guaranteed near-optimal policy, and guaranteed model accuracy.
  arXiv  Detail & Related papers  (2023-07-01T18:35:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.