Robust Exploration in Directed Controller Synthesis via Reinforcement Learning with Soft Mixture-of-Experts

• URL: http://arxiv.org/abs/2602.19244v1
• Date: Sun, 22 Feb 2026 15:56:13 GMT
• Title: Robust Exploration in Directed Controller Synthesis via Reinforcement Learning with Soft Mixture-of-Experts
• Abstract summary: On-the-fly Directed Controller Synthesis (OTFDCS) mitigates state-space explosion by incrementally exploring the system.<n>Recent reinforcement learning approaches learn such policies and achieve promising zero-shot generalization from small training instances to larger unseen ones.<n>We propose a Soft Mixture-of-Experts framework that combines multiple RL experts via a prior-confidence gating mechanism and treats these anisotropic behaviors as complementary specializations.
• Score: 2.9451141736604396
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-the-fly Directed Controller Synthesis (OTF-DCS) mitigates state-space explosion by incrementally exploring the system and relies critically on an exploration policy to guide search efficiently. Recent reinforcement learning (RL) approaches learn such policies and achieve promising zero-shot generalization from small training instances to larger unseen ones. However, a fundamental limitation is anisotropic generalization, where an RL policy exhibits strong performance only in a specific region of the domain-parameter space while remaining fragile elsewhere due to training stochasticity and trajectory-dependent bias. To address this, we propose a Soft Mixture-of-Experts framework that combines multiple RL experts via a prior-confidence gating mechanism and treats these anisotropic behaviors as complementary specializations. The evaluation on the Air Traffic benchmark shows that Soft-MoE substantially expands the solvable parameter space and improves robustness compared to any single expert.

Related papers

• SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models [51.99912169291891]
  Reinforcement learning (RL) has emerged as a promising paradigm for training reasoning-oriented models.<n>We propose a training paradigm that interleaves conventional RL with inverse reinforcement learning (IRL)<n>Our findings suggest that alternating between RL and IRL offers an effective pathway toward extending the exploration capacity of reasoning-oriented large language models.
  arXiv  Detail & Related papers  (2026-04-18T13:49:47Z)
• S-GRPO: Unified Post-Training for Large Vision-Language Models [11.901218794806796]
  We propose Supervised Group Relative Policy Optimization (S-GRPO) for adapting Large Vision-Language Models (LVLMs)<n>S-GRPO integrates the guidance of imitation learning into the multi-trajectory exploration of preference optimization.<n>It gracefully bridges the gap between Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL)
  arXiv  Detail & Related papers  (2026-04-17T08:39:07Z)
• Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning [10.4404175363648]
  Post-training paradigms for Large Language Models (LLMs) face a fundamental dilemma.<n>SFT provides stability (low variance) but suffers from high fitting bias, while RL enables exploration (low bias) but grapples with high gradient variance.<n>Existing unified optimization strategies often employ naive loss weighting, overlooking the statistical conflict between these distinct gradient signals.<n>We propose textbfDYPO (Dynamic Policy Optimization), a unified framework designed to structurally mitigate this conflict.
  arXiv  Detail & Related papers  (2026-04-10T03:42:16Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration [56.074760766965085]
  PRISM achieves a dynamics-aware framework that arbitrates data based on its degree of cognitive conflict with the model's existing knowledge.<n>Our findings suggest that disentangling data based on internal optimization regimes is crucial for scalable and robust agent alignment.
  arXiv  Detail & Related papers  (2026-01-12T05:43:20Z)
• Sample-Efficient Neurosymbolic Deep Reinforcement Learning [49.60927398960061]
  We propose a neuro-symbolic Deep RL approach that integrates background symbolic knowledge to improve sample efficiency.<n>Online reasoning is performed to guide the training process through two mechanisms.<n>We show improved performance over a state-of-the-art reward machine baseline.
  arXiv  Detail & Related papers  (2026-01-06T09:28:53Z)
• Stratified GRPO: Handling Structural Heterogeneity in Reinforcement Learning of LLM Search Agents [90.45197506653341]
  Large language model (LLM) agents increasingly rely on external tools such as search engines to solve complex, multi-step problems.<n>The trajectories of search agents are structurally heterogeneous, where variations in the number, placement, and outcomes of search calls lead to fundamentally different answer directions and reward distributions.<n>Standard policy gradient methods, which use a single global baseline, suffer from what we identify and formalize as cross-stratum bias.<n>We propose Stratified GRPO, whose central component, Stratified Advantage Normalization (SAN), partitions trajectories into homogeneous strata based on their structural properties and computes advantages locally within each
  arXiv  Detail & Related papers  (2025-10-07T17:59:13Z)
• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting [91.38734024438357]
  Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) are two prominent post-training paradigms for refining the capabilities and aligning the behavior of Large Language Models (LLMs)<n>Existing approaches that integrate SFT and RL often face the risk of disrupting established response patterns and inducing overfitting to expert data.<n>We propose CHORD, a framework for Controllable Harmonization of On- and Off-Policy Reinforcement Learning via Dynamic Weighting.
  arXiv  Detail & Related papers  (2025-08-15T11:20:03Z)
• Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning [53.85659415230589]
  This paper systematically reviews widely adoptedReinforcement learning techniques.<n>We present clear guidelines for selecting RL techniques tailored to specific setups.<n>We also reveal that a minimalist combination of two techniques can unlock the learning capability of critic-free policies.
  arXiv  Detail & Related papers  (2025-08-11T17:39:45Z)
• On the Effect of Regularization in Policy Mirror Descent [0.0]
  Policy Mirror Descent (PMD) has emerged as a unifying framework in reinforcement learning (RL)<n>PMD incorporates two key regularization components: (i) a distance term that enforces a trust region for stable policy updates and (ii) an MDP regularizer that augments the reward function to promote structure and robustness.<n>This work provides a large-scale empirical analysis of the interplay between these two regularization techniques, running over 500k training seeds on small RL environments.
  arXiv  Detail & Related papers  (2025-07-11T16:19:45Z)
• DPO: A Differential and Pointwise Control Approach to Reinforcement Learning [3.2857981869020327]
  Reinforcement learning (RL) in continuous state-action spaces remains challenging in scientific computing.<n>We introduce Differential Reinforcement Learning (Differential RL), a novel framework that reformulates RL from a continuous-time control perspective.<n>We develop Differential Policy Optimization (DPO), a pointwise, stage-wise algorithm that refines local movement operators.
  arXiv  Detail & Related papers  (2024-04-24T03:11:12Z)
• Multi-fidelity reinforcement learning framework for shape optimization [0.8258451067861933]
  We introduce a controlled transfer learning framework that leverages a multi-fidelity simulation setting. Our strategy is deployed for an airfoil shape optimization problem at high Reynolds numbers. Our results demonstrate this framework's applicability to other scientific DRL scenarios.
  arXiv  Detail & Related papers  (2022-02-22T20:44:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.