Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap

• URL: http://arxiv.org/abs/2605.16349v1
• Date: Fri, 08 May 2026 04:17:10 GMT
• Title: Geometric Asymmetry in MoE Specialization: Functional Decorrelation and Representational Overlap
• Abstract summary: Mixture-of-Experts (MoE) architectures achieve scalable capacity through sparse routing.<n>We introduce a unified Jacobian-PCA-Grassmann framework for analyzing MoE layers in both function space and representation space.
• Score: 0.5414847001704249
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) architectures achieve scalable capacity through sparse routing, yet the geometric structure of expert specialization remains poorly understood. We introduce a unified Jacobian-PCA-Grassmann framework for analyzing MoE layers in both function space and representation space. Across pretrained MoE Transformers (Mistral, Qwen), we find a consistent structural asymmetry: experts exhibit strong functional decorrelation (consistently low, near-zero cross-expert Jacobian alignment) while their routed representations occupy distinct but partially overlapping subspaces. This indicates that functional decorrelation and representation overlap coexist rather than coincide in MoE specialization. Controlled routing experiments further indicate that routing sparsity appears to be a key factor shaping this geometry: top-k routing induces sharper functional separation and larger subspace divergence, whereas fully soft routing yields more entangled expert structure. Together, these results suggest a geometric interpretation in which MoE layers may be viewed as implementing locally decorrelated operators over overlapping submanifolds on a shared representation manifold, and provide a general diagnostic framework for studying conditional computation in modern Transformer architectures.

Related papers

• Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers [4.71326501896672]
  Recent state-space models (SSMs) such as Mamba achieve language modeling performance comparable to transformers.<n>This raises an important question: how do these structural differences influence the geometry and functional nature of their internal representations?<n>We study this question through a multi-scale analysis of representations in transformers, SSMs, and hybrid architecture.
  arXiv  Detail & Related papers  (2026-09-08T12:57:30Z)
• Structured Learning on Mapper Representations [0.18682913407152388]
  The Mapper algorithm from topological data analysis (TDA) provides a different perspective by decomposing data into overlapping local regions connected through a nerve construction.<n>We develop a framework for learning over Mapper induced structured representations.<n>We study mathematical properties of these representations, including invariance under relabeling, a distance functional on the space of Mapper representations, structural complexity of multiscale decompositions, and learning oriented stability under representation perturbations.
  arXiv  Detail & Related papers  (2026-08-22T17:15:37Z)
• Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression [68.1291440835878]
  decompositions have emerged as a promising direction, offering compact parameterizations well suited to Transformer weight structures.<n>Existing studies evaluate these methods in narrow settings, leaving unclear whether tensorization is effective at large-scale deployment.<n>We systematically evaluate tensor compression across dense and MoE, establishing performance trade-offs grounded in both empirical analysis and theoretical analysis.
  arXiv  Detail & Related papers  (2026-06-02T10:45:21Z)
• Are Common Substructures Transferable? Riemannian Graph Foundation Model with Neural Vector Bundles [53.65187206795911]
  We develop a graph intrinsic geometry learning framework called Neural Vector Bundle.<n>Building on this, we design GAUGE, a pretrainable neural architecture that constructs the vector bundle and flattens geometrically compatible local coordinates.<n>We empirically validate its superior expressiveness in challenging tasks including zero-shot link prediction and graph isomorphism.
  arXiv  Detail & Related papers  (2026-06-02T07:35:42Z)
• Uncovering the Latent Potential of Deep Intermediate Representations [16.988545433223933]
  We show that task-relevant information is distributed non-monotonically across layers and cannot be recovered by nave aggregation.<n>We introduce Layer-wise Optimal Embedding Selection (LOES), a constructive spectral method that identifies task-discriminative subspaces.<n>Our method reveals how semantic factors are distributed across layers, thereby enabling cross-lingual and cross-modal interpretability analyses.
  arXiv  Detail & Related papers  (2026-05-21T20:58:42Z)
• GeoMamba: A Geometry-driven MambaVision Framework and Dataset for Fine-grained Optical-SAR Object Retrieval [54.741349848771144]
  GeoMamba is a geometry-driven framework for optical-SAR fine-grained retrieval.<n>GFI module enhances cross-modal feature interaction and incorporates structural priors.<n>GeoMamba outperforms existing methods, achieving 63.3% mAP and 77.0% Rank-1 accuracy in all-to-all retrieval setting.
  arXiv  Detail & Related papers  (2026-05-19T12:08:09Z)
• Beyond Point-Wise Matching: Structural Representation Alignment for Accelerating Diffusion Transformers [93.3976834364707]
  We propose sREPA, a structural REPresentation Alignment framework to enforce consistency in the relational geometry of feature maps.<n>By encouraging the model to internalize holistic spatial layouts and structural correlations from pre-trained features, sREPA achieves faster and more stable convergence.
  arXiv  Detail & Related papers  (2026-05-16T12:01:04Z)
• Every Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexity [15.786915433105868]
  We show that a broad class of feedforward neural networks learn (have finite sample complexity) in the PAC model.<n>Our results reposition finite-sample PAC learnability as a baseline rather than a differentiator.
  arXiv  Detail & Related papers  (2026-05-08T01:26:25Z)
• GRIFDIR: Graph Resolution-Invariant FEM Diffusion Models in Function Spaces over Irregular Domains [56.121725064621295]
  Score-based diffusion models in function spaces offer a principled framework for modelling function-valued data.<n>However, practical implementations have struggled to fully realise these benefits.<n>We propose a novel architecture that represents generalised convolutional kernels as finite graph kernels.
  arXiv  Detail & Related papers  (2026-05-05T08:33:52Z)
• Hyperbolic Enhanced Representation Learning for Incomplete Multi-view Clustering [57.38215918201251]
  We propose HERL, a Hyperbolic Enhanced Representation Learning framework for Incomplete Multi-View Clustering.<n> operating within the Poincaré ball, HERL constructs a structure-aware latent space to enhance representation learning.<n>We show that HERL consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-04-18T10:50:46Z)
• Geometry-Preserving Aggregation for Mixture-of-Experts Embedding Models [4.125187280299246]
  Mixture-of-Experts (MoE) embedding models combine expert outputs using weighted linear summation, implicitly assuming a linear subspace structure in the embedding space.<n> Geometric analysis of a modern MoE embedding model reveals that expert outputs lie on a shared hyperspherical manifold characterized by tightly concentrated norms and substantial angular separation.<n>Spherical Barycentric Aggregation (SBA) is introduced as a geometry-preserving aggregation operator that separates radial and angular components to maintain hyperspherical structure while remaining fully compatible with existing routing mechanisms.
  arXiv  Detail & Related papers  (2026-02-15T08:00:56Z)
• Mixture-of-Experts as Soft Clustering: A Dual Jacobian-PCA Spectral Geometry Perspective [0.5414847001704249]
  Mixture-of-Experts (MoE) architectures are commonly motivated by efficiency and conditional computation.<n>We study MoEs through a geometric lens, interpreting routing as a form of soft partitioning of the representation space into overlapping local charts.
  arXiv  Detail & Related papers  (2026-01-09T23:07:14Z)
• Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits [22.333229451408414]
  Transformer-based language models exhibit complex and distributed behavior, yet their internal computations remain poorly understood.<n>Existing interpretability methods treat attention heads and multilayer perceptron layers (MLPs) as indivisible units, overlooking possibilities of functional substructure learned within them.<n>We introduce a more fine-grained perspective that decomposes these components into singular directions, revealing superposed and independent computations within a single head or mechanistic.
  arXiv  Detail & Related papers  (2025-11-25T12:59:15Z)
• GeoGNN: Quantifying and Mitigating Semantic Drift in Text-Attributed Graphs [59.61242815508687]
  Graph neural networks (GNNs) on text--attributed graphs (TAGs) encode node texts using pretrained language models (PLMs) and propagate these embeddings through linear neighborhood aggregation.<n>This work introduces a local PCA-based metric that measures the degree of semantic drift and provides the first quantitative framework to analyze how different aggregation mechanisms affect manifold structure.
  arXiv  Detail & Related papers  (2025-11-12T06:48:43Z)
• The Neural Differential Manifold: An Architecture with Explicit Geometric Structure [8.201374511929538]
  This paper introduces the Neural Differential Manifold (NDM), a novel neural network architecture that explicitly incorporates geometric structure into its fundamental design.<n>We analyze the theoretical advantages of this approach, including its potential for more efficient optimization, enhanced continual learning, and applications in scientific discovery and controllable generative modeling.
  arXiv  Detail & Related papers  (2025-10-29T02:24:27Z)
• Generalized Linear Mode Connectivity for Transformers [87.32299363530996]
  A striking phenomenon is linear mode connectivity (LMC), where independently trained models can be connected by low- or zero-loss paths.<n>Prior work has predominantly focused on neuron re-ordering through permutations, but such approaches are limited in scope.<n>We introduce a unified framework that captures four symmetry classes: permutations, semi-permutations, transformations, and general invertible maps.<n>This generalization enables, for the first time, the discovery of low- and zero-barrier linear paths between independently trained Vision Transformers and GPT-2 models.
  arXiv  Detail & Related papers  (2025-06-28T01:46:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.