SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks

• URL: http://arxiv.org/abs/2412.13053v1
• Date: Tue, 17 Dec 2024 16:15:04 GMT
• Title: SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks
• Authors: Mátyás Vincze, Laura Ferrarotti, Leonardo Lucio Custode, Bruno Lepri, Giovanni Iacca,
• Abstract summary: SMOSE is a novel method to train sparsely activated interpretable controllers. It combines a set of interpretable decisionmakers, trained to be experts in different basic skills, and an interpretable router that assigns tasks among the experts. We then distill decision trees from the weights of the router, significantly improving the ease of interpretation.
• Score: 6.408395876568997
• License:
• Abstract: Continuous control tasks often involve high-dimensional, dynamic, and non-linear environments. State-of-the-art performance in these tasks is achieved through complex closed-box policies that are effective, but suffer from an inherent opacity. Interpretable policies, while generally underperforming compared to their closed-box counterparts, advantageously facilitate transparent decision-making within automated systems. Hence, their usage is often essential for diagnosing and mitigating errors, supporting ethical and legal accountability, and fostering trust among stakeholders. In this paper, we propose SMOSE, a novel method to train sparsely activated interpretable controllers, based on a top-1 Mixture-of-Experts architecture. SMOSE combines a set of interpretable decisionmakers, trained to be experts in different basic skills, and an interpretable router that assigns tasks among the experts. The training is carried out via state-of-the-art Reinforcement Learning algorithms, exploiting load-balancing techniques to ensure fair expert usage. We then distill decision trees from the weights of the router, significantly improving the ease of interpretation. We evaluate SMOSE on six benchmark environments from MuJoCo: our method outperforms recent interpretable baselines and narrows the gap with noninterpretable state-of-the-art algorithms

Related papers

• Interactive Agents to Overcome Ambiguity in Software Engineering [61.40183840499932]
  AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions. Making unwarranted assumptions and failing to ask clarifying questions can lead to suboptimal outcomes. We study the ability of LLM agents to handle ambiguous instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance.
  arXiv  Detail & Related papers  (2025-02-18T17:12:26Z)
• On the effectiveness of discrete representations in sparse mixture of experts [33.809432499123275]
  We propose a new architecture dubbed Vector-Quantized Mixture of Experts (VQMoE) VQMoE is an effective solution for scaling up model capacity without increasing the computational costs. We show that VQMoE achieves a 28% improvement in routers compared to other SMoE routing methods.
  arXiv  Detail & Related papers  (2024-11-28T22:32:01Z)
• Complexity Experts are Task-Discriminative Learners for Any Image Restoration [80.46313715427928]
  We introduce complexity experts" -- flexible expert blocks with varying computational complexity and receptive fields. This preference effectively drives task-specific allocation, assigning tasks to experts with the appropriate complexity. The proposed MoCE-IR model outperforms state-of-the-art methods, affirming its efficiency and practical applicability.
  arXiv  Detail & Related papers  (2024-11-27T15:58:07Z)
• Designing an Interpretable Interface for Contextual Bandits [0.0]
  We design a new interface to explain to domain experts the underlying behaviour of a bandit. Our findings suggest that by carefully balancing technical rigour with accessible presentation, it is possible to empower non-experts to manage complex machine learning systems.
  arXiv  Detail & Related papers  (2024-09-23T15:47:44Z)
• Model-Agnostic Interpretation Framework in Machine Learning: A Comparative Study in NBA Sports [0.2937071029942259]
  We propose an innovative framework to reconcile the trade-off between model performance and interpretability. Our approach is centered around modular operations on high-dimensional data, which enable end-to-end processing while preserving interpretability. We have extensively tested our framework and validated its superior efficacy in achieving a balance between computational efficiency and interpretability.
  arXiv  Detail & Related papers  (2024-01-05T04:25:21Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Imitation from Observation With Bootstrapped Contrastive Learning [12.048166025000976]
  Imitation from observation (IfO) is a learning paradigm that consists of training autonomous agents in a Markov Decision Process. We present BootIfOL, an IfO algorithm that aims to learn a reward function that takes an agent trajectory and compares it to an expert. We evaluate our approach on a variety of control tasks showing that we can train effective policies using a limited number of demonstrative trajectories.
  arXiv  Detail & Related papers  (2023-02-13T17:32:17Z)
• Hierarchical Skills for Efficient Exploration [70.62309286348057]
  In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration. Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design. We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
  arXiv  Detail & Related papers  (2021-10-20T22:29:32Z)
• Mixture of Robust Experts (MoRE): A Flexible Defense Against Multiple Perturbations [33.868833849116044]
  A Mixture of Robust Experts (MoRE) approach enables flexible integration of a broad range of robust experts with superior performance. In this work, through a gating mechanism, we assemble a set of expert networks either adversarially trained to deal with a particular perturbation type or normally trained for boosting accuracy on clean data.
  arXiv  Detail & Related papers  (2021-04-21T15:27:07Z)
• Closing the Closed-Loop Distribution Shift in Safe Imitation Learning [80.05727171757454]
  We treat safe optimization-based control strategies as experts in an imitation learning problem. We train a learned policy that can be cheaply evaluated at run-time and that provably satisfies the same safety guarantees as the expert.
  arXiv  Detail & Related papers  (2021-02-18T05:11:41Z)
• An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction [84.49035467829819]
  We show that it is possible to better manage this trade-off by optimizing a bound on the Information Bottleneck (IB) objective. Our fully unsupervised approach jointly learns an explainer that predicts sparse binary masks over sentences, and an end-task predictor that considers only the extracted rationale.
  arXiv  Detail & Related papers  (2020-05-01T23:26:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.