Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts

• URL: http://arxiv.org/abs/2606.19036v1
• Date: Wed, 17 Jun 2026 13:06:00 GMT
• Title: Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts
• Abstract summary: Sparse Mixture-of-Experts (SMoE) architectures are now widely deployed in language and vision models.<n>Top-$k$ expert selection that enables conditional routing renders the SMoE map inherently discontinuous.<n>We show that lower-order discontinuity sets dominate, whereas higher-order ones occupy a vanishingly small relative volume.<n>We propose a simple smoothing mechanism that can be applied to existing SMoEs, incorporating softly experts near discontinuities.
• Score: 15.710271326191219
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse Mixture-of-Experts (SMoE) architectures are now widely deployed in state-of-the-art language and vision models, where conditional routing allows scaling to very large networks. However, this very Top-$k$ expert selection that enables conditional routing also renders the SMoE map inherently discontinuous. In the vicinity of these discontinuity surfaces, even inputs that are arbitrarily close may activate substantially different sets of experts resulting in significantly different outputs. In this work we give a rigorous geometric and stochastic analysis of these discontinuities. We first classify them by order, determined by the number of tied experts at a switching event. Using measure-theoretic slicing arguments, we establish asymptotic volume estimates for the thickened discontinuity surfaces, showing that lower-order discontinuity sets dominate, whereas higher-order ones occupy a vanishingly small relative volume. Next, modeling random perturbations in the input space via a diffusion process, we prove that the path eventually encounter a discontinuity, and moreover that the first hit almost surely occurs on an order-1 discontinuity with explicit finite-time probability bounds. We further derive occupation-time bounds that quantify the duration the random path spend in the neighborhoods of each discontinuity order. These theoretical results imply that inputs are more likely to lie near lower order discontinuities. Motivated by this insight, we propose a simple smoothing mechanism that can be directly applied to existing SMoEs, softly incorporating experts near discontinuities; our analysis guarantees that the added computational overhead remains small while providing localized smoothing near discontinuities, and experiments across language and vision tasks show that smoothing not only enforces continuity of the SMoE map but also enhances empirical performance.

Related papers

• Scale-Consistent Posterior Dynamics for Diffusion Inverse Problems [16.685352002333225]
  Posterior sampling with a pretrained diffusion prior is governed by a conditional score whose likelihood component is generally intractable.<n>We show that a noise-conditioned path whose targets approach the clean posterior produces a continuous surrogate SDE.<n>Experiments on FFHQ and ImageNet with 100 score evaluations demonstrate competitive reconstruction fidelity for superresolution and deblurring.
  arXiv  Detail & Related papers  (2026-08-15T09:38:41Z)
• Fitting Unknown Number of Hyperplanes with Manifold Optimization [57.48093263119306]
  Fitting unknown number of linearplanes to data is a fundamental yet challenging problem in machine learning.<n>Existing approaches often struggle with optimum optima or lack geometric consistency.
  arXiv  Detail & Related papers  (2026-05-27T14:02:20Z)
• Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics [92.39053980710702]
  Gradient Descent (SGD) is commonly modeled as a Langevin process, assuming that minibatch noise acts as Brownian motion.<n>This approximation relies on a continuous-time limit and a sqrt(eta) noise scaling that does not match the discrete SGD update at finite learning rate.<n>We propose an alternative formulation of SGD as deterministic dynamics in a fluctuating loss landscape induced by minibatch sampling.
  arXiv  Detail & Related papers  (2026-05-21T15:50:40Z)
• Reading Calibrated Uncertainty from Language Model Trajectories [46.663987199083245]
  Methods that probe the model's internal activations feed raw hidden states into opaque snapshots and leave implicit the layer-wise trajectory by which a representation is formed.<n>We extract eleven scale-invariant geometric features, tracing the cumulative path of per-layer updates, and feed them to a sparse linear probe.<n>The probe outperforms MSP under selective abstention, with baseline scaling with up to 21 AURC points.
  arXiv  Detail & Related papers  (2026-05-19T19:24:29Z)
• A Differentiable Bayesian Relaxation for Latent Partial-Order Inference [2.124421328820064]
  Many ranking and agent trace datasets are recorded as linear orders even though their latent structure is only partially ordered.<n>We introduce a differentiable relaxation for latent partial-order inference from such traces.<n>We prove soft transitivity, sharp-limit frontier recovery, and convergence to the hard likelihood.
  arXiv  Detail & Related papers  (2026-05-07T21:47:41Z)
• On the continuity of flows [0.10152838128195464]
  We show that the optimal velocity field under flow matching objectives may exhibit spatial discontinuities.<n>We argue that this discontinuity arises from the requirement that continuous flows must bifurcate to map a single mode to multiple modes.<n>Our analysis suggests that this phenomenon is not specific to $L2$ loss, but rather may be a consequence of topological mismatch between distributions.
  arXiv  Detail & Related papers  (2025-12-14T20:00:39Z)
• Defect Bootstrap: Tight Ground State Bounds in Spontaneous Symmetry Breaking Phases [0.0]
  bootstrap methods have enabled rigorous two-sided bounds on local observables directly in the thermodynamic limit.<n>These bounds inevitably become loose in symmetry broken phases, where local constraints are insufficient to capture long-range order.<n>We introduce a $textitdefect bootstrap$ framework that resolves this limitation by embedding the system into an auxiliary $textitdefect model.<n>Our results demonstrate that physically motivated constraint sets can dramatically enhance the power of bootstrap methods for quantum many-body systems.
  arXiv  Detail & Related papers  (2025-11-25T21:17:54Z)
• Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations [57.179679246370114]
  We identify the distribution of random perturbations that minimizes the estimator's variance as the perturbation stepsize tends to zero.<n>Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length.
  arXiv  Detail & Related papers  (2025-10-22T19:06:39Z)
• Flow based approach for Dynamic Temporal Causal models with non-Gaussian or Heteroscedastic Noises [37.02662517645979]
  We introduce FANTOM, a unified framework for causal discovery.<n>It handles non-stationary processes along with non-Gaussian and heteroscedastic noises.<n>It simultaneously infers the number of regimes and their corresponding indices and learns each regime's Directed Acyclic Graph.
  arXiv  Detail & Related papers  (2025-06-20T15:12:43Z)
• Discrete-to-Continuum Approach for the Analytic Continuation of One-Particle Propagator on the Circle [0.0]
  We derive finite expressions for the free discrete propagator on a circle suitable to numerical evaluation.<n>These expressions allow for the reconstruction of the propagator in the continuous circle limit.<n>We show that the well-known infinite line limit is consistently recovered within this approach.
  arXiv  Detail & Related papers  (2025-03-18T18:01:48Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective [49.17352150219212]
  Federated AveragingFedAvg, also known as Local SGD, is one of the most popular algorithms in Federated Learning (FL) We show how to analyze this quantity from the Differential Equation (SDE) perspective.
  arXiv  Detail & Related papers  (2021-11-05T22:16:11Z)
• Lifting the Convex Conjugate in Lagrangian Relaxations: A Tractable Approach for Continuous Markov Random Fields [53.31927549039624]
  We show that a piecewise discretization preserves better contrast from existing discretization problems. We apply this theory to the problem of matching two images.
  arXiv  Detail & Related papers  (2021-07-13T12:31:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.