'Explaining RL Decisions with Trajectories': A Reproducibility Study

• URL: http://arxiv.org/abs/2411.07200v1
• Date: Mon, 11 Nov 2024 18:24:27 GMT
• Title: 'Explaining RL Decisions with Trajectories': A Reproducibility Study
• Abstract summary: Original paper introduces a novel approach in explainable reinforcement learning based on the attribution decisions of an agent to specific clusters of trajectories encountered during training. We verify the main claims from the paper, which state that (i) training on less trajectories induces a lower initial state value, (ii) trajectories in a cluster present similar high-level patterns, (iii) distant trajectories influence the decision of an agent, and (iv) humans correctly identify the attributed trajectories to the decision of the agent.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work investigates the reproducibility of the paper 'Explaining RL decisions with trajectories'. The original paper introduces a novel approach in explainable reinforcement learning based on the attribution decisions of an agent to specific clusters of trajectories encountered during training. We verify the main claims from the paper, which state that (i) training on less trajectories induces a lower initial state value, (ii) trajectories in a cluster present similar high-level patterns, (iii) distant trajectories influence the decision of an agent, and (iv) humans correctly identify the attributed trajectories to the decision of the agent. We recover the environments used by the authors based on the partial original code they provided for one of the environments (Grid-World), and implemented the remaining from scratch (Seaquest, HalfCheetah, Breakout and Q*Bert). While we confirm that (i), (ii), and (iii) partially hold, we extend on the largely qualitative experiments from the authors by introducing a quantitative metric to further support (iii), and new experiments and visual results for (i). Moreover, we investigate the use of different clustering algorithms and encoder architectures to further support (ii). We could not support (iv), given the limited extent of the original experiments. We conclude that, while some of the claims can be supported, further investigations and experiments could be of interest. We recognise the novelty of the work from the authors and hope that our work paves the way for clearer and more transparent approaches.

Related papers

• Long-Horizon Autonomous Architecture Research with a Language-Model Agent: A Behavioural Case Study [0.3683202928838613]
  We study what happens when a single general-purpose large language model acts as the sole researcher on a long-horizon neural architecture design problem.<n>The study comprises three phases, separated by human-declared transitions, that progressively expand the agent's tool surface or problem scale.<n>We conclude that workflow design was at least as influential as agent capability in this study and propose diversified search, budgeted moonshot hypotheses, explicit forks, and regime-aware re-validation.
  arXiv  Detail & Related papers  (2026-08-03T09:56:34Z)
• Combining Trained Models in Reinforcement Learning [0.0]
  Deep reinforcement learning (DRL) has delivered strong results in domains such as Atari and Go, but it still suffers from high sample cost and weak transfer beyond the training setting.<n>A common response is to reuse information from previously trained models through transfer, distillation, ensemble methods, or federated training.<n>This paper presents a PRISMA-guided systematic review of empirical studies on pretrained knowledge reuse in DRL.
  arXiv  Detail & Related papers  (2026-05-04T02:37:55Z)
• Structure Liberates: How Constrained Sensemaking Produces More Novel Research Output [23.77139747975822]
  We introduce SCISENSE, a framework that operationalizes ideation as a structured sequence of eight cognitive stages.<n>We show that Target-trained models achieve a 2.0% improvement in trajectory quality over Infer-trained models.<n>This suggests that targeted ideation reduces cognitive burden on downstream agents, freeing them to explore more creatively.
  arXiv  Detail & Related papers  (2026-05-01T10:50:59Z)
• SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models [51.99912169291891]
  Reinforcement learning (RL) has emerged as a promising paradigm for training reasoning-oriented models.<n>We propose a training paradigm that interleaves conventional RL with inverse reinforcement learning (IRL)<n>Our findings suggest that alternating between RL and IRL offers an effective pathway toward extending the exploration capacity of reasoning-oriented large language models.
  arXiv  Detail & Related papers  (2026-04-18T13:49:47Z)
• Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration [49.9937230730202]
  We propose Search-R2, a novel Actor-Refiner collaboration framework that enhances reasoning through targeted intervention.<n>Our approach decomposes the generation process into an Actor, which produces initial reasoning trajectories.<n>We show that Search-R2 consistently outperforms strong RAG and RL-based baselines across model scales.
  arXiv  Detail & Related papers  (2026-02-03T15:32:09Z)
• Beneficial Reasoning Behaviors in Agentic Search and Effective Post-training to Obtain Them [23.986035712600657]
  We propose a reasoning-driven pipeline to study effective reasoning behavior patterns in agentic search.<n>We identify four beneficial reasoning behaviors: Information Verification, Authority Evaluation, Adaptive Search, and Error Recovery.<n>We show that behavior priming yields over 35% gains in Llama3.2-3B and Qwen3-1.7B compared to directly training agentic search models with RL.
  arXiv  Detail & Related papers  (2025-10-08T00:20:35Z)
• Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory? [13.980638430366625]
  Reasoning LLMs are trained to verbalize their reasoning process, yielding strong gains on complex tasks.<n>A key prerequisite is the ability to assess the usefulness and build on another model's partial thinking.<n>This paper investigates the question: can standard solo-reasoning training pipelines deliver desired off-trajectory behaviors?
  arXiv  Detail & Related papers  (2025-10-07T19:42:50Z)
• A Practitioner's Guide to Multi-turn Agentic Reinforcement Learning [12.179148605060298]
  We study what actually works and what doesn't for training large language models as agents via multi-turn reinforcement learning.<n>We break down the design space into three inter-related pillars -- environment, reward, and policy.<n>We distill these findings into a training recipe that guides co-design across the three pillars.
  arXiv  Detail & Related papers  (2025-10-01T17:23:04Z)
• A Tale of Two Experts: Cooperative Learning for Source-Free Unsupervised Domain Adaptation [59.88864205383671]
  Source-Free Unsupervised Domain Adaptation (SFUDA) addresses the realistic challenge of adapting a source-trained model to a target domain without access to the source data.<n>Existing SFUDA methods either exploit only the source model's predictions or fine-tune large multimodal models.<n>We propose the Experts Cooperative Learning (EXCL) to exploit complementary insights and the latent structure of target data.
  arXiv  Detail & Related papers  (2025-09-26T11:39:50Z)
• Reasoning with Exploration: An Entropy Perspective on Reinforcement Learning for LLMs [112.40801692473723]
  Balancing exploration and exploitation is a central goal in reinforcement learning (RL)<n>We introduce a minimal modification to standard RL with only one line of code: augmenting the advantage function with an entropy-based term.<n>Our method achieves significant gains on the Pass@K metric, even when evaluated with extremely large K values.
  arXiv  Detail & Related papers  (2025-06-17T17:54:03Z)
• Enhancing Study-Level Inference from Clinical Trial Papers via RL-based Numeric Reasoning [10.449112615828419]
  We conceptualise the problem as one of quantitative reasoning.<n>We develop a numeric reasoning system composed of a numeric data extraction model and an effect estimate component.
  arXiv  Detail & Related papers  (2025-05-28T22:59:45Z)
• Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoning [55.36978389831446]
  We recast reflective exploration within the Bayes-Adaptive RL framework.<n>Our resulting algorithm, BARL, instructs the LLM to stitch and switch strategies based on observed outcomes.
  arXiv  Detail & Related papers  (2025-05-26T22:51:00Z)
• Fusing Bidirectional Chains of Thought and Reward Mechanisms A Method for Enhancing Question-Answering Capabilities of Large Language Models for Chinese Intangible Cultural Heritage [3.7756107931620666]
  We propose a novel training method that integrates a bidirectional chains of thought and a reward mechanism.<n>This method is built upon ICH-Qwen, a large language model specifically designed for the field of intangible cultural heritage.
  arXiv  Detail & Related papers  (2025-05-13T02:05:25Z)
• Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data [3.376269351435396]
  We develop a formal perspective on probing using structural causal models (SCM) We extend a recent study of LMs in the context of a synthetic grid-world navigation task. Our techniques provide robust empirical evidence for the ability of LMs to induce the latent concepts underlying text.
  arXiv  Detail & Related papers  (2024-07-18T17:59:27Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• Knowledge Transfer across Multiple Principal Component Analysis Studies [8.602833477729899]
  We propose a two-step transfer learning algorithm to extract useful information from multiple source principal component analysis (PCA) studies. In the first step, we integrate the shared subspace information across multiple studies by a proposed method named as Grassmannian barycenter. The resulting estimator for the shared subspace from the first step is further utilized to estimate the target private subspace.
  arXiv  Detail & Related papers  (2024-03-12T09:15:12Z)
• STEERING: Stein Information Directed Exploration for Model-Based Reinforcement Learning [111.75423966239092]
  We propose an exploration incentive in terms of the integral probability metric (IPM) between a current estimate of the transition model and the unknown optimal. Based on KSD, we develop a novel algorithm algo: textbfSTEin information dirtextbfEcted exploration for model-based textbfReinforcement LearntextbfING.
  arXiv  Detail & Related papers  (2023-01-28T00:49:28Z)
• Back-to-Bones: Rediscovering the Role of Backbones in Domain Generalization [1.6799377888527687]
  Domain Generalization studies the capability of a deep learning model to generalize to out-of-training distributions. Recent research has provided a reproducible benchmark for DG, pointing out the effectiveness of naive empirical risk minimization (ERM) over existing algorithms. In this paper, we evaluate the backbones proposing a comprehensive analysis of their intrinsic generalization capabilities.
  arXiv  Detail & Related papers  (2022-09-02T15:30:17Z)
• Provable Benefits of Representational Transfer in Reinforcement Learning [59.712501044999875]
  We study the problem of representational transfer in RL, where an agent first pretrains in a number of source tasks to discover a shared representation. We show that given generative access to source tasks, we can discover a representation, using which subsequent linear RL techniques quickly converge to a near-optimal policy.
  arXiv  Detail & Related papers  (2022-05-29T04:31:29Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)
• Abstract, Rationale, Stance: A Joint Model for Scientific Claim Verification [18.330265729989843]
  We propose an approach, named as ARSJoint, that jointly learns the modules for the three tasks with a machine reading comprehension framework. The experimental results on the benchmark dataset SciFact show that our approach outperforms the existing works.
  arXiv  Detail & Related papers  (2021-09-13T10:07:26Z)
• Interpretable performance analysis towards offline reinforcement learning: A dataset perspective [6.526790418943535]
  We propose a two-fold taxonomy for existing offline RL algorithms. We explore the correlation between the performance of different types of algorithms and the distribution of actions under states. We create a benchmark platform on the Atari domain, entitled easy go (RLEG), at an estimated cost of more than 0.3 million dollars.
  arXiv  Detail & Related papers  (2021-05-12T07:17:06Z)
• Unsupervised Transfer Learning for Spatiotemporal Predictive Networks [90.67309545798224]
  We study how to transfer knowledge from a zoo of unsupervisedly learned models towards another network. Our motivation is that models are expected to understand complex dynamics from different sources. Our approach yields significant improvements on three benchmarks fortemporal prediction, and benefits the target even from less relevant ones.
  arXiv  Detail & Related papers  (2020-09-24T15:40:55Z)
• When Relation Networks meet GANs: Relation GANs with Triplet Loss [110.7572918636599]
  Training stability is still a lingering concern of generative adversarial networks (GANs) In this paper, we explore a relation network architecture for the discriminator and design a triplet loss which performs better generalization and stability. Experiments on benchmark datasets show that the proposed relation discriminator and new loss can provide significant improvement on variable vision tasks.
  arXiv  Detail & Related papers  (2020-02-24T11:35:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.