More Experts, Worse Dynamics: Inverse Scaling and Spectral Bias in Mixture-of-Experts State-Space Models

• URL: http://arxiv.org/abs/2608.21840v1
• Date: Sat, 22 Aug 2026 08:22:32 GMT
• Title: More Experts, Worse Dynamics: Inverse Scaling and Spectral Bias in Mixture-of-Experts State-Space Models
• Abstract summary: We study a next-step prediction task on sequences composed of three regimes.<n>Operator-level mixture models consistently fail to outperform a single-expert baseline.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixture-of-Experts (MoE) architectures are commonly motivated as a way to increase expressivity by decomposing complex systems into simpler local dynamics. This intuition has recently been extended to spectral state-space models, where mixing stable operators is assumed to enable adaptation to heterogeneous or regime-switching time series. We critically evaluate this assumption in a controlled synthetic setting designed to isolate dynamical rather than representational challenges. We study a next-step prediction task on sequences composed of three regimes: chaotic dynamics generated by the Mackey-Glass system, a stable oscillatory regime, and a noise-dominated autoregressive regime. Across extensive ablations including capacity scaling, oracle routing, frozen-expert variants, and comparisons to output-level MoE baselines, operator-level mixture models consistently fail to outperform a single-expert baseline. Increasing the number of experts leads to inverse scaling, routing collapses or fails to induce meaningful specialization, and even perfect regime supervision does not prevent degradation in global performance. Furthermore, we show that apparent improvements in mean squared error on chaotic trajectories can be misleading. Phase-space analysis reveals that lower error often arises from temporal smoothing that destroys the geometry of the underlying attractor rather than from faithful modeling of the dynamics. These results identify a likely limitation of operator interpolation under the studied parameterization and training protocol, and underscore the need for geometry-aware evaluation when assessing regime-switching dynamical systems.

Related papers

• DAW: Dynamics-Aware Weighting for Deep Learning Forecasts of Chaotic Systems [7.490065985639902]
  We introduce Dynamics-Aware Weighting (DAW), a data-centric objective reweighting framework.<n>On the chaotic KS equation, DAW consistently outperforms uniform training, purely statistical density weighting, and its randomly permuted ablation.
  arXiv  Detail & Related papers  (2026-08-23T08:07:03Z)
• Learning Spatio-Temporal Foundation Models from Pure Synthetic Data [85.23980441738811]
  NeoST is a scalable synthetic-training corpus to real-world bias, a latent-space reasoning architecture that generates and iteratively refines multiple future trajectories, and latent-space objectives that emphasize structural dynamics and enable inference-time correction under distribution shifts.<n>Extensive experiments across diverse real-world benchmarks show that NeoST consistently outperforms existing STFMs in diverse real-worldtemporal systems.
  arXiv  Detail & Related papers  (2026-06-27T02:56:46Z)
• A Quadratic Order Reduction -- Gaussian Process Ordinary Differential Equation framework for the inference of Large Continuous Dynamical Systems [0.0]
  We propose a framework for forecasting complex dynamical systems via a kernel autonomous ordinary differential equation approach.<n>We integrate it with quadratic order reduced-order modelling and sphere projection for learning the latent dynamics efficiently while preserving stability.<n>Results demonstrate the potential of the framework as a robust and stable tool for forecasting complex dynamical systems with rigorous uncertainty quantification.
  arXiv  Detail & Related papers  (2026-06-11T08:49:18Z)
• PnP-Corrector: A Universal Correction Framework for Coupled Spatiotemporal Forecasting [62.22445401483844]
  Existing methods are severely constrained by the bottleneck of compounding errors.<n>We propose a universal framework called -Cast-Corrector (Plug-and-Play Corrector)<n>Our method significantly enhances the long-term stability of coupled forecasting systems.
  arXiv  Detail & Related papers  (2026-05-09T13:12:33Z)
• KoopGen: Koopman Generator Networks for Representing and Predicting Dynamical Systems with Continuous Spectra [65.11254608352982]
  We introduce a generator-based neural Koopman framework that models dynamics through a structured, state-dependent representation of Koopman generators.<n>By exploiting the intrinsic Cartesian decomposition into skew-adjoint and self-adjoint components, KoopGen separates conservative transport from irreversible dissipation.
  arXiv  Detail & Related papers  (2026-02-15T06:32:23Z)
• A Mechanistic Analysis of Transformers for Dynamical Systems [4.590170084532207]
  We study the representational capabilities and limitations of single-layer Transformers when applied to dynamical data.<n>For linear systems, we show that the convexity constraint imposed by softmax attention fundamentally restricts the class of dynamics that can be represented.<n>For nonlinear systems under partial observability, attention instead acts as an adaptive delay-embedding mechanism.
  arXiv  Detail & Related papers  (2025-12-24T11:21:07Z)
• AERMANI-Diffusion: Regime-Conditioned Diffusion for Dynamics Learning in Aerial Manipulators [2.376629619475859]
  Aerial manipulators undergo rapid, configuration-dependent changes in inertial coupling forces and aerodynamic forces.<n>We propose a regime-conditioned diffusion framework that models the full distribution of residual forces using a conditional diffusion process and a lightweight temporal encoder.
  arXiv  Detail & Related papers  (2025-12-11T16:10:32Z)
• Drift No More? Context Equilibria in Multi-Turn LLM Interactions [58.69551510148673]
  contexts drift is the gradual divergence of a model's outputs from goal-consistent behavior across turns.<n>Unlike single-turn errors, drift unfolds temporally and is poorly captured by static evaluation metrics.<n>We show that multi-turn drift can be understood as a controllable equilibrium phenomenon rather than as inevitable decay.
  arXiv  Detail & Related papers  (2025-10-09T04:48:49Z)
• Elucidated Rolling Diffusion Models for Probabilistic Weather Forecasting [52.6508222408558]
  We introduce Elucidated Rolling Diffusion Models (ERDM)<n>ERDM is the first framework to unify a rolling forecast structure with the principled, performant design of Elucidated Diffusion Models (EDM)<n>On 2D Navier-Stokes simulations and ERA5 global weather forecasting at 1.5circ resolution, ERDM consistently outperforms key diffusion-based baselines.
  arXiv  Detail & Related papers  (2025-06-24T21:44:31Z)
• Predicting Forced Responses of Probability Distributions via the Fluctuation-Dissipation Theorem and Generative Modeling [0.0]
  We present a data-driven framework for estimating the response of higher-order moments of nonlinear systems to small external perturbations.<n>We combine GFDT with score-based generative modeling to estimate the system's score function directly from data.<n>Our method is validated on several models relevant to climate dynamics.
  arXiv  Detail & Related papers  (2025-04-17T20:54:33Z)
• Stochastically forced ensemble dynamic mode decomposition for forecasting and analysis of near-periodic systems [65.44033635330604]
  We introduce a novel load forecasting method in which observed dynamics are modeled as a forced linear system. We show that its use of intrinsic linear dynamics offers a number of desirable properties in terms of interpretability and parsimony. Results are presented for a test case using load data from an electrical grid.
  arXiv  Detail & Related papers  (2020-10-08T20:25:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.