From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning

• URL: http://arxiv.org/abs/2605.23382v1
• Date: Fri, 22 May 2026 08:50:55 GMT
• Title: From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning
• Abstract summary: We propose a unified personalized Agentic RL framework that embeds personalization into training-time optimization.<n>Our framework consistently outperforms strong memory and RL baselines.
• Score: 15.774638671062393
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic reinforcement learning (Agentic RL) has achieved strong progress in tasks with clear success signals. However, many real-world agent applications require user-conditioned behavior: the same query may call for different planning strategies and tool-use decisions across users. This setting raises key challenges: generic rewards cannot capture heterogeneous user preferences, observed behaviors are entangled with conformity effects, and flat memories cannot support personalized skill retrieval. To this end, we propose a unified personalized Agentic RL framework that embeds personalization into training-time optimization. At its core is \emph{Personalized Anchor Reward-Decoupled Policy Optimization} (\textbf{PARPO}), which decouples generic task-quality rewards from personalized preference rewards and uses user-specific anchors to stabilize learning under heterogeneous reward scales. We further introduce a two-stage preference-disentangled reward model and \emph{Preference-Aligned Skill Evolution Graph Memory} (\textbf{PSGM}) for personalized supervision and preference-aligned skill retrieval. Together, they form a closed loop of preference identification, policy optimization, and structured skill accumulation. Experiments on ETAPP, ETAPP-Hard, and SJAgent show that our framework consistently outperforms strong memory and RL baselines. Code and data are included in the supplementary materials.

Related papers

• From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents [56.187590188698955]
  Large Language Models have enabled increasingly capable autonomous agents, yet personalization remains critical for making such agents practically useful.<n>Recent benchmarks have begun evaluating personalization in agents, but they largely rely on static preference snapshots, fixed interaction logs, or question answering over predefined user profiles.<n>We introduce IBA-Bench, a benchmark for implicit behavioral alignment constructed from longitudinal interaction histories.<n>We propose IBA-Agent, an agent framework that reconciles conflicting priorities through broad retrieval and trajectory-level alignment.
  arXiv  Detail & Related papers  (2026-08-03T12:52:04Z)
• ODYSSE: Episode-wise Policy Optimization for Personalized Agentic Reasoning [48.692251671993574]
  We present ODYSSE, a framework for personalized agentic reasoning.<n>ODYSSE addresses long action horizons and strong cross-step dependencies in personalized agentic reasoning.<n>We show that ODYSSE consistently outperforms both specialist and general-purpose LVLMs.
  arXiv  Detail & Related papers  (2026-07-28T07:25:04Z)
• Beyond One-Size-Fits-All: Adaptive Test-Time Augmentation for Sequential Recommendation [3.873910497341511]
  Test-time augmentation has become a promising approach for mitigating data sparsity in sequential recommendation.<n>Existing TTA methods typically rely on uniform, user-agnostic augmentation strategies.<n>AdaTTA learns to select sequence-specific augmentation operators on a per-sequence basis.
  arXiv  Detail & Related papers  (2026-04-17T14:56:29Z)
• FedPDPO: Federated Personalized Direct Preference Optimization for Large Language Model Alignment [55.97027207627]
  Fed PDPO (Federated Personalized Direct Preference Optimization) is a personalized framework for preference alignment of large language models (LLMs)<n>It adopts a parameter-efficient fine-tuning architecture where each client maintains a frozen pretrained LLM backbone augmented with a Low-Rank Adaptation (LoRA) adapter, enabling communication-efficient aggregation.<n>Experiments on multiple preference datasets demonstrate state-of-the-art performance, achieving up to 4.80% average accuracy improvements in federated intra-domain and cross-domain settings.
  arXiv  Detail & Related papers  (2026-03-20T08:24:49Z)
• Synthetic Interaction Data for Scalable Personalization in Large Language Models [67.31884245564086]
  We introduce a high-fidelity synthetic data generation framework called PersonaGym.<n>Unlike prior work that treats personalization as static persona-preference pairs, PersonaGym models a dynamic preference process.<n>We release PersonaAtlas, a large-scale, high-quality, and diverse synthetic dataset of high-fidelity multi-turn personalized interaction trajectories.
  arXiv  Detail & Related papers  (2026-02-12T20:41:22Z)
• FedUTR: Federated Recommendation with Augmented Universal Textual Representation for Sparse Interaction Scenarios [58.33672784127872]
  Federated recommendations (FRs) have emerged as an on-device privacy-preserving paradigm.<n>We propose a novel method named FedUTR, which incorporates item textual representations as a complement to interaction behaviors.<n>Our method achieves superior performance, with improvements of up to 59% across all datasets compared to the SOTA baselines.
  arXiv  Detail & Related papers  (2026-01-29T08:55:19Z)
• One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment [55.86333374784959]
  We argue that addressing these constraints requires a paradigm shift from fitting data to learn user preferences to learn the process of preference adaptation.<n>We propose Meta Reward Modeling (MRM), which reformulates personalized reward modeling as a meta-learning problem.<n>We show that MRM enhances few-shot personalization, improves user robustness, and consistently outperforms baselines.
  arXiv  Detail & Related papers  (2026-01-26T17:55:52Z)
• Optimizing User Profiles via Contextual Bandits for Retrieval-Augmented LLM Personalization [27.490675380289318]
  We argue that relevance serves as an unreliable proxy for utility.<n>We propose PURPLE, a contextual bandit framework that oPtimizes UseR Profiles for Llm pErsonalization.<n>In contrast to a greedy selection of the most relevant records, PURPLE treats profile construction as a set generation process.
  arXiv  Detail & Related papers  (2026-01-17T15:05:36Z)
• MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization [56.074760766965085]
  Group-Relative Policy Optimization has emerged as an efficient paradigm for aligning Large Language Models (LLMs)<n>We propose MAESTRO, which treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck.<n>We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal.
  arXiv  Detail & Related papers  (2026-01-12T05:02:48Z)
• Proxy Model-Guided Reinforcement Learning for Client Selection in Federated Recommendation [41.93915547763831]
  Federated recommender systems have emerged as a promising privacy-preserving paradigm, enabling personalized recommendation services without exposing users' raw data.<n>Most existing FedRS frameworks adopt fully random client selection strategy in each training round.<n>We propose ProxyRL-FRS, a proxy model-guided reinforcement learning framework tailored for client selection in federated recommendation.
  arXiv  Detail & Related papers  (2025-08-14T07:03:39Z)
• LoRe: Personalizing LLMs via Low-Rank Reward Modeling [47.12507639759984]
  We introduce a novel framework that leverages low-rank preference modeling to efficiently learn and generalize user-specific reward functions.<n>We validate our method on multiple preference datasets, demonstrating superior generalization to unseen users and improved accuracy in preference prediction tasks.
  arXiv  Detail & Related papers  (2025-04-20T01:16:24Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications.<n> Ensuring their alignment with the diverse preferences of individual users has become a critical challenge.<n>We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.