TREX: Trajectory Explanations for Multi-Objective Reinforcement Learning

• URL: http://arxiv.org/abs/2603.21988v1
• Date: Mon, 23 Mar 2026 13:55:14 GMT
• Title: TREX: Trajectory Explanations for Multi-Objective Reinforcement Learning
• Abstract summary: Trajectory based Explainability framework to explain Multi-objective Reinforcement Learning policies.<n>Trajectory based Explainability framework to explain Multi-objective Reinforcement Learning policies.
• Score: 1.2922946578413577
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning (RL) has demonstrated its ability to solve complex decision-making problems in a variety of domains, by optimizing reward signals obtained through interaction with an environment. However, many real-world scenarios involve multiple, potentially conflicting objectives that cannot be easily represented by a single scalar reward. Multi-Objective Reinforcement Learning (MORL) addresses this limitation by enabling agents to optimize several objectives simultaneously, explicitly reasoning about trade-offs between them. However, the ``black box" nature of the RL models makes the decision process behind chosen objective trade-offs unclear. Current Explainable Reinforcement Learning (XRL) methods are typically designed for single scalar rewards and do not account for explanations with respect to distinct objectives or user preferences. To address this gap, in this paper we propose TREX, a Trajectory based Explainability framework to explain Multi-objective Reinforcement Learning policies, based on trajectory attribution. TREX generates trajectories directly from the learned expert policy, across different user preferences and clusters them into semantically meaningful temporal segments. We quantify the influence of these behavioural segments on the Pareto trade-off by training complementary policies that exclude specific clusters, measuring the resulting relative deviation on the observed rewards and actions compared to the original expert policy. Experiments on multi-objective MuJoCo environments - HalfCheetah, Ant and Swimmer, demonstrate the framework's ability to isolate and quantify the specific behavioural patterns.

Related papers

• LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback [4.210862582629798]
  We present LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback.<n>It is a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies.<n>Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives.
  arXiv  Detail & Related papers  (2026-07-31T15:50:29Z)
• Objective-Behavior Alignment: Diagnostics for MORL Policy Selection [9.112571355018035]
  Multi-objective reinforcement learning produces sets of policies that explicitly represent trade-offs between objectives.<n>We propose an exploratory diagnostic workflow that automatically highlights behavioral variation along the Pareto front.<n>We validate our approach on simple grid examples and scale it to continuous control benchmarks, demonstrating that it remains effective as problem complexity increases.
  arXiv  Detail & Related papers  (2026-06-19T11:10:26Z)
• Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitation [49.86232017439639]
  We introduce Multi-Output Augmented Behavioral Cloning (MA-BC)<n>MA-BC partitions divergent expert data while pooling state-action pairs where no behavior conflict is observed.<n>We establish a novel lower bound for multi-objective imitation learning, demonstrating that MA-BC is minimax optimal.
  arXiv  Detail & Related papers  (2026-05-12T11:49:08Z)
• AIPO: Learning to Reason from Active Interaction [54.10819421625103]
  AIPO enables the policy model to proactively consult three functional collaborative agents when encountering reasoning bottlenecks.<n>AIPO consistently improves reasoning performance, generalizes robustly across different policy models and RLVR algorithms, and effectively expands the reasoning capability boundary of the policy model.
  arXiv  Detail & Related papers  (2026-05-08T19:06:55Z)
• Multi-Objective Constraint Inference using Inverse reinforcement learning [4.8029888032686445]
  Multi-Objective Constraint Inference (MOCI) is a novel framework designed to jointly extract shared constraints and individual preferences from heterogeneous expert trajectories.<n>MOCI effectively models and learns from diverse, and potentially conflicting, behaviors.
  arXiv  Detail & Related papers  (2026-05-07T21:09:09Z)
• TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training [53.93696896939915]
  Training tool-use agents typically rely on Supervised Fine-Tuning (SFT) on successful trajectories and Reinforcement Learning (RL) on pass-rate-selected tasks.<n>We propose TopoCurate, an interaction-aware framework that projects multi-trial rollouts from the same task into a unified semantic quotient topology.<n>TopoCurate achieves consistent gains of 4.2% (SFT) and 6.9% (RL) over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-02T10:38:54Z)
• Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback [14.81819959351561]
  We show that multi-objective RLAIF can produce policies that yield balanced trade-offs reflecting different user priorities without laborious reward engineering.<n>We argue that integrating RLAIF into multi-objective RL offers a scalable path toward user-aligned policy learning.
  arXiv  Detail & Related papers  (2026-02-24T09:47:25Z)
• SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning [50.93295951454092]
  We introduce a set level diversity objective defined over sampled trajectories using kernelized similarity.<n>Our approach derives a leave-one-out marginal contribution for each sampled trajectory and integrates this objective as a plug-in advantage shaping term for policy optimization.<n>Experiments across a range of model scales demonstrate the effectiveness of our proposed algorithm, consistently outperforming strong baselines in both Pass@1 and Pass@K across various benchmarks.
  arXiv  Detail & Related papers  (2026-02-01T07:13:20Z)
• From Sparse Decisions to Dense Reasoning: A Multi-attribute Trajectory Paradigm for Multimodal Moderation [59.27094165576015]
  We propose a novel learning paradigm (UniMod) that transitions from sparse decision-making to dense reasoning traces.<n>By constructing structured trajectories encompassing evidence grounding, modality assessment, risk mapping, policy decision, and response generation, we reformulate monolithic decision tasks into a multi-dimensional boundary learning process.<n>We introduce specialized optimization strategies to decouple task-specific parameters and rebalance training dynamics, effectively resolving interference between diverse objectives in multi-task learning.
  arXiv  Detail & Related papers  (2026-01-28T09:29:40Z)
• Multi-Path Collaborative Reasoning via Reinforcement Learning [54.8518809800168]
  Chain-of-Thought (CoT) reasoning has significantly advanced the problem-solving capabilities of Large Language Models (LLMs)<n>Recent methods attempt to address this by generating soft abstract tokens to enable reasoning in a continuous semantic space.<n>We propose Multi-Path Perception Policy Optimization (M3PO), a novel reinforcement learning framework that explicitly injects collective insights into the reasoning process.
  arXiv  Detail & Related papers  (2025-12-01T10:05:46Z)
• Behaviour Discovery and Attribution for Explainable Reinforcement Learning [6.123880364445758]
  Building trust in reinforcement learning (RL) agents requires understanding why they make certain decisions.<n>Existing explainability methods often focus on single states or entire trajectories.<n>We propose a fully offline, reward-free framework for behavior discovery and segmentation.
  arXiv  Detail & Related papers  (2025-03-19T08:06:00Z)
• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process.<n>We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)
• Pareto Inverse Reinforcement Learning for Diverse Expert Policy Generation [6.876580618014666]
  In this paper, we adapt inverse reinforcement learning (IRL) by using reward distance estimates for regularizing the discriminator. We show that ParIRL outperforms other IRL algorithms for various multi-objective control tasks.
  arXiv  Detail & Related papers  (2024-08-22T03:51:39Z)
• Value function interference and greedy action selection in value-based multi-objective reinforcement learning [1.4206639868377509]
  Multi-objective reinforcement learning (MORL) algorithms extend conventional reinforcement learning (RL) We show that, if the user's utility function maps widely varying vector-values to similar levels of utility, this can lead to interference. We demonstrate empirically that avoiding the use of random tie-breaking when identifying greedy actions can ameliorate, but not fully overcome, the problems caused by value function interference.
  arXiv  Detail & Related papers  (2024-02-09T09:28:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.