SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks

• URL: http://arxiv.org/abs/2412.13053v1
• Date: Tue, 17 Dec 2024 16:15:04 GMT
• Title: SMOSE: Sparse Mixture of Shallow Experts for Interpretable Reinforcement Learning in Continuous Control Tasks
• Abstract summary: SMOSE is a novel method to train sparsely activated interpretable controllers.<n>It combines a set of interpretable decisionmakers, trained to be experts in different basic skills, and an interpretable router that assigns tasks among the experts.<n>We then distill decision trees from the weights of the router, significantly improving the ease of interpretation.
• Score: 6.408395876568997
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continuous control tasks often involve high-dimensional, dynamic, and non-linear environments. State-of-the-art performance in these tasks is achieved through complex closed-box policies that are effective, but suffer from an inherent opacity. Interpretable policies, while generally underperforming compared to their closed-box counterparts, advantageously facilitate transparent decision-making within automated systems. Hence, their usage is often essential for diagnosing and mitigating errors, supporting ethical and legal accountability, and fostering trust among stakeholders. In this paper, we propose SMOSE, a novel method to train sparsely activated interpretable controllers, based on a top-1 Mixture-of-Experts architecture. SMOSE combines a set of interpretable decisionmakers, trained to be experts in different basic skills, and an interpretable router that assigns tasks among the experts. The training is carried out via state-of-the-art Reinforcement Learning algorithms, exploiting load-balancing techniques to ensure fair expert usage. We then distill decision trees from the weights of the router, significantly improving the ease of interpretation. We evaluate SMOSE on six benchmark environments from MuJoCo: our method outperforms recent interpretable baselines and narrows the gap with noninterpretable state-of-the-art algorithms

Related papers

• Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills [53.15216425100295]
  We introduce Skill Self-Play (Skill-SP), a co-evolutionary framework comprising a proposer, a solver, and a skill controller.<n>The proposer generates challenging tasks conditioned on dynamically sampled skills; the solver explores candidate solutions to push its capability boundaries; and the skill controller collects execution feedback to update and expand the skill library.<n>This interactive co-evolution effectively bridges the gap between structured verification and open-ended exploration.
  arXiv  Detail & Related papers  (2026-07-24T17:59:22Z)
• Emergent Compositional Skills in Mixture-of-Experts VLAs [4.404201871360367]
  We ask whether a VLA trained with a simplified Mixture-of-Experts (MoE) action head can emergently learn to decompose tasks into reusable, interpretable primitives.<n>We find that learned experts are heavily reused across tasks and consistently correspond to qualitatively distinct low-level behaviors.
  arXiv  Detail & Related papers  (2026-07-22T22:36:52Z)
• Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning [10.01449025634975]
  Existing methods treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities.<n>We introduce Mixture of Sparse Experts for Task Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition.<n>We show SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines.
  arXiv  Detail & Related papers  (2026-06-05T17:53:52Z)
• Steering LLMs via Scalable Interactive Oversight [74.12746881843044]
  Large Language Models increasingly automate complex, long-horizon tasks such as emphvibe coding, a supervision gap has emerged.<n>It presents a critical challenge in scalable oversight: enabling humans to responsibly steer AI systems on tasks that surpass their own ability to specify or verify.
  arXiv  Detail & Related papers  (2026-02-04T04:52:00Z)
• SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning [83.66308307152808]
  We propose StAbilized Mixture-of-Experts (SAME) for Multimodal Continual Instruction Tuning (MCIT)<n>SAME stabilizes expert selection by decomposing routing dynamics into subspaces and updating only task-relevant directions.<n>It also introduces adaptive expert activation to freeze selected experts during training, reducing redundant and cross-task interference.
  arXiv  Detail & Related papers  (2026-02-02T11:47:06Z)
• Split-on-Share: Mixture of Sparse Experts for Task-Agnostic Continual Learning [10.01449025634975]
  Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma.<n>We introduce SETA, a framework that resolves the plasticity-stability conflict by decomposing the model into modular subspaces.<n>We show that SETA consistently outperforms state-of-the-art parameter-efficient fine-tuning-based continual learning methods.
  arXiv  Detail & Related papers  (2026-01-24T22:39:22Z)
• SelfAI: Building a Self-Training AI System with LLM Agents [79.10991818561907]
  SelfAI is a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations.<n>An Experiment Manager orchestrates parallel, fault-tolerant training across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback.<n>Across regression, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials.
  arXiv  Detail & Related papers  (2025-11-29T09:18:39Z)
• Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencoder [59.89996751196727]
  Sparse autoencoders (SAEs) have emerged as a powerful tool for interpreting large language models.<n>SAEs' hidden layers have high dimensionality to satisfy sparsity constraints, resulting in prohibitive training and inference costs.<n>Recent Mixture of Experts (MoE) approaches attempt to address this by SAEs into narrower expert networks with gated activation.<n>We propose two key innovations: (1) Multiple Expert Activation that simultaneously engages semantically weighted expert subsets to encourage specialization, and (2) Feature Scaling that enhances diversity through adaptive high-frequency scaling.
  arXiv  Detail & Related papers  (2025-11-07T22:19:34Z)
• BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning [82.925106913459]
  Reinforcement finetuning (RFT) is a key technique for aligning Large Language Models (LLMs) with human preferences and enhancing reasoning.<n>We introduce BOTS, a unified framework for Bayesian Online Task Selection in RFT reinforcement finetuning.
  arXiv  Detail & Related papers  (2025-10-30T11:15:23Z)
• Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts [19.707274733121412]
  Sparse Mixture of Experts (sMoE) has become a pivotal approach for scaling large vision-language models.<n>We propose Input Domain Aware MoE, a novel routing framework that leverages a probabilistic mixture model to better partition the input space.<n>By modeling routing probabilities as a mixture of distributions, our method enables experts to develop clear specialization boundaries while achieving balanced utilization.
  arXiv  Detail & Related papers  (2025-10-18T11:01:03Z)
• MoRE-LLM: Mixture of Rule Experts Guided by a Large Language Model [54.14155564592936]
  We propose a Mixture of Rule Experts guided by a Large Language Model (MoRE-LLM) MoRE-LLM steers the discovery of local rule-based surrogates during training and their utilization for the classification task. LLM is responsible for enhancing the domain knowledge alignment of the rules by correcting and contextualizing them.
  arXiv  Detail & Related papers  (2025-03-26T11:09:21Z)
• Interactive Agents to Overcome Ambiguity in Software Engineering [61.40183840499932]
  AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions. Making unwarranted assumptions and failing to ask clarifying questions can lead to suboptimal outcomes. We study the ability of LLM agents to handle ambiguous instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance.
  arXiv  Detail & Related papers  (2025-02-18T17:12:26Z)
• On the effectiveness of discrete representations in sparse mixture of experts [33.809432499123275]
  We propose a new architecture dubbed Vector-Quantized Mixture of Experts (VQMoE)<n>VQMoE is an effective solution for scaling up model capacity without increasing the computational costs.<n>We show that VQMoE achieves a 28% improvement in routers compared to other SMoE routing methods.
  arXiv  Detail & Related papers  (2024-11-28T22:32:01Z)
• Complexity Experts are Task-Discriminative Learners for Any Image Restoration [80.46313715427928]
  We introduce complexity experts" -- flexible expert blocks with varying computational complexity and receptive fields.<n>This preference effectively drives task-specific allocation, assigning tasks to experts with the appropriate complexity.<n>The proposed MoCE-IR model outperforms state-of-the-art methods, affirming its efficiency and practical applicability.
  arXiv  Detail & Related papers  (2024-11-27T15:58:07Z)
• Designing an Interpretable Interface for Contextual Bandits [0.0]
  We design a new interface to explain to domain experts the underlying behaviour of a bandit. Our findings suggest that by carefully balancing technical rigour with accessible presentation, it is possible to empower non-experts to manage complex machine learning systems.
  arXiv  Detail & Related papers  (2024-09-23T15:47:44Z)
• RILe: Reinforced Imitation Learning [60.63173816209543]
  RILe is a novel trainer-student system that learns a dynamic reward function based on the student's performance and alignment with expert demonstrations. RILe enables better performance in complex settings where traditional methods falter, outperforming existing methods by 2x in complex simulated robot-locomotion tasks.
  arXiv  Detail & Related papers  (2024-06-12T17:56:31Z)
• Model-Agnostic Interpretation Framework in Machine Learning: A Comparative Study in NBA Sports [0.2937071029942259]
  We propose an innovative framework to reconcile the trade-off between model performance and interpretability. Our approach is centered around modular operations on high-dimensional data, which enable end-to-end processing while preserving interpretability. We have extensively tested our framework and validated its superior efficacy in achieving a balance between computational efficiency and interpretability.
  arXiv  Detail & Related papers  (2024-01-05T04:25:21Z)
• RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
  We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning. Our proposed method uses reinforcement learning with user intervention signals themselves as rewards. This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
  arXiv  Detail & Related papers  (2023-11-21T21:05:21Z)
• Imitation from Observation With Bootstrapped Contrastive Learning [12.048166025000976]
  Imitation from observation (IfO) is a learning paradigm that consists of training autonomous agents in a Markov Decision Process. We present BootIfOL, an IfO algorithm that aims to learn a reward function that takes an agent trajectory and compares it to an expert. We evaluate our approach on a variety of control tasks showing that we can train effective policies using a limited number of demonstrative trajectories.
  arXiv  Detail & Related papers  (2023-02-13T17:32:17Z)
• Mixture of Robust Experts (MoRE): A Flexible Defense Against Multiple Perturbations [33.868833849116044]
  A Mixture of Robust Experts (MoRE) approach enables flexible integration of a broad range of robust experts with superior performance. In this work, through a gating mechanism, we assemble a set of expert networks either adversarially trained to deal with a particular perturbation type or normally trained for boosting accuracy on clean data.
  arXiv  Detail & Related papers  (2021-04-21T15:27:07Z)
• Closing the Closed-Loop Distribution Shift in Safe Imitation Learning [80.05727171757454]
  We treat safe optimization-based control strategies as experts in an imitation learning problem. We train a learned policy that can be cheaply evaluated at run-time and that provably satisfies the same safety guarantees as the expert.
  arXiv  Detail & Related papers  (2021-02-18T05:11:41Z)
• An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction [84.49035467829819]
  We show that it is possible to better manage this trade-off by optimizing a bound on the Information Bottleneck (IB) objective. Our fully unsupervised approach jointly learns an explainer that predicts sparse binary masks over sentences, and an end-task predictor that considers only the extracted rationale.
  arXiv  Detail & Related papers  (2020-05-01T23:26:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.