Boundary Mass and the Soft-to-Hard Limit in Mixture-of-Experts

• URL: http://arxiv.org/abs/2605.02124v1
• Date: Mon, 04 May 2026 01:07:21 GMT
• Title: Boundary Mass and the Soft-to-Hard Limit in Mixture-of-Experts
• Abstract summary: We study the singularity at the population level for squared-loss MoE regression.<n>The central object is the emphboundary mass, namely the probability that the top two router scores are separated by only a small margin.<n>We show that the zero-temperature limit is controlled by a thin geometric layer around routing interfaces, not by the full input space.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Softmax-routed mixture-of-experts models approach hard routing as the temperature tends to zero, but this limit is singular near routing ties. This paper studies that singularity at the population level for squared-loss MoE regression. The central object is the \emph{boundary mass}, namely the probability that the top two router scores are separated by only a small margin. Under smoothness and transversality assumptions on the router and input law, we prove coarea/tube estimates showing that this mass is linear in the slab width, with leading constant given by a surface integral over the routing interface in the binary case. These estimates yield quantitative soft-to-hard risk bounds and, under compactness and uniform margin control, $Γ$-convergence of the soft objectives to the hard-routing objective. The main conclusion is that the zero-temperature limit is controlled by a thin geometric layer around routing interfaces, not by the full input space. We then use this geometric core in two more model-dependent directions. In a teacher--student setting, we prove a conditional landscape-transfer principle showing that, when the profiled hard-routing problem has favorable identifiability and curvature and the relevant derivatives transfer at boundary-layer scale, small-temperature soft routing inherits approximate teacher recovery and strict-saddle behavior away from teacher-equivalent partitions. We also give a reduced two-expert Gaussian calculation that illustrates a local symmetry-breaking mechanism aligned with the teacher separator.

Related papers

• A Minimal Bifurcation Model of Load Imbalance in a Softmax Mixture-of-Experts Router [0.0]
  We propose a minimal dynamical model of adaptive softmax routing for a two-expert Mixture-of-Experts layer.<n>We derive exact parametric equations for the bifurcation set and the local normal form of the cusp catastrophe.
  arXiv  Detail & Related papers  (2026-05-27T21:29:30Z)
• Fitting Unknown Number of Hyperplanes with Manifold Optimization [57.48093263119306]
  Fitting unknown number of linearplanes to data is a fundamental yet challenging problem in machine learning.<n>Existing approaches often struggle with optimum optima or lack geometric consistency.
  arXiv  Detail & Related papers  (2026-05-27T14:02:20Z)
• Boundary Geometry Turns Entanglement into Steering [5.478582839093083]
  Quantum entanglement does not necessarily imply Einstein-Podolsky-Rosen steering.<n>We identify a emphboundary mechanism that closes this gap when an entangled state meets the boundary of the trusted state space.<n>We provide a filtered standard form, an explicit rank-three Cholesky parametrization, and a compact boundary certificate.
  arXiv  Detail & Related papers  (2026-05-20T14:33:16Z)
• A meshfree exterior calculus for generalizable and data-efficient learning of physics from point clouds [8.15594797043476]
  We introduce a meshfree exterior calculus (MEEC) for learning structure-preserving descriptions of physics on point clouds.<n>We use it to build MEEC-Net, a data-efficient surrogate that transfers across resolutions, geometries, and physical parameters.
  arXiv  Detail & Related papers  (2026-05-08T19:59:40Z)
• The Geometric Cost of Normalization: Affine Bounds on the Bayesian Complexity of Neural Networks [0.0]
  We prove that LayerNorm's mean-centering step, by confining data to a linear hyperplane (through the origin), reduces the Local Learning Coefficient (LLC) of the subsequent weight matrix by exactly $m/2$.<n>We further show that Softmax simplex data introduces a "smuggled bias" that activates the same $m/2$ LLC drop when paired with an explicit downstream bias.
  arXiv  Detail & Related papers  (2026-03-28T22:15:45Z)
• Riemannian Flow Matching for Disentangled Graph Domain Adaptation [51.98961391065951]
  Graph Domain Adaptation (GDA) typically uses adversarial learning to align graph embeddings in Euclidean space.<n>DisRFM is a geometry-aware GDA framework that unifies embedding and flow-based transport.
  arXiv  Detail & Related papers  (2026-01-31T11:05:35Z)
• Exact Constraint Enforcement in Physics-Informed Extreme Learning Machines using Null-Space Projection Framework [0.0]
  This work introduces Null-Space Projected PIELM, achieving exact constraint enforcement through algebraic projection in coefficient space.<n>NP-PIELM exploits the geometric structure of the admissible coefficient manifold, recognizing that it admits a decomposition through the null space of the boundary operator.<n>This eliminates penalty coefficients, dual variables, and problem-specific constructions while preserving single-shot training efficiency.
  arXiv  Detail & Related papers  (2026-01-16T05:18:56Z)
• Optimal Boundary Control of Diffusion on Graphs via Linear Programming [2.064612766965483]
  We propose a framework for steady-state diffusion and flux optimization on geometric networks.<n> Boundary potentials act as controls that drive interior flux according to a linear network Laplacian.<n>The analysis connects classical results such as the Minkowski--Weyl decomposition, Hoffman's bound, and the fundamental theorem of linear programming with modern network-based diffusion modeling.
  arXiv  Detail & Related papers  (2025-11-05T02:41:07Z)
• Global $\mathcal{L}^2$ minimization at uniform exponential rate via geometrically adapted gradient descent in Deep Learning [1.4050802766699084]
  We consider the scenario of supervised learning in Deep Learning (DL) networks.<n>We choose the gradient flow with respect to the Euclidean metric in the output layer of the DL network.
  arXiv  Detail & Related papers  (2023-11-27T02:12:02Z)
• $\texttt{FedBC}$: Calibrating Global and Local Models via Federated Learning Beyond Consensus [66.62731854746856]
  In federated learning (FL), the objective of collaboratively learning a global model through aggregation of model updates across devices tends to oppose the goal of personalization via local information. In this work, we calibrate this tradeoff in a quantitative manner through a multi-criterion-based optimization. We demonstrate that $texttFedBC$ balances the global and local model test accuracy metrics across a suite datasets.
  arXiv  Detail & Related papers  (2022-06-22T02:42:04Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)
• Zero Pixel Directional Boundary by Vector Transform [77.63061686394038]
  We re-interpret boundaries as 1-D surfaces and formulate a one-to-one vector transform function that allows for training of boundary prediction completely avoiding the class imbalance issue. Our problem formulation leads to the estimation of direction as well as richer contextual information of the boundary, and, if desired, the availability of zero-pixel thin boundaries also at training time.
  arXiv  Detail & Related papers  (2022-03-16T17:55:31Z)
• Deep Learning Approximation of Diffeomorphisms via Linear-Control Systems [91.3755431537592]
  We consider a control system of the form $dot x = sum_i=1lF_i(x)u_i$, with linear dependence in the controls. We use the corresponding flow to approximate the action of a diffeomorphism on a compact ensemble of points.
  arXiv  Detail & Related papers  (2021-10-24T08:57:46Z)
• Lifting the Convex Conjugate in Lagrangian Relaxations: A Tractable Approach for Continuous Markov Random Fields [53.31927549039624]
  We show that a piecewise discretization preserves better contrast from existing discretization problems. We apply this theory to the problem of matching two images.
  arXiv  Detail & Related papers  (2021-07-13T12:31:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.