From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts

• URL: http://arxiv.org/abs/2609.06403v1
• Date: Sun, 06 Sep 2026 05:43:22 GMT
• Title: From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts
• Abstract summary: Test-time scaling produces cohorts of reasoning rollouts, yet there is no standard label-free account of how their internal computation reorganizes as inference unfolds.<n>We introduce routing effective rank deff, the entropy-effective dimensionality of a cross-rollout graph built from MoE expert-routing similarity.
• Score: 61.695980784491475
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Test-time scaling produces cohorts of reasoning rollouts, yet there is no standard label-free account of how their internal computation reorganizes as inference unfolds. We introduce routing effective rank deff, the entropy-effective dimensionality of a cross-rollout graph built from MoE expert-routing similarity. Across ten MoE configurations and five math/science benchmarks, deff exhibits a reproducible low-high-low trajectory, with a prominent interior maximum in 98.5% of 3,105 model-question cohorts: routing similarity is concentrated early, maximally differentiated at intermediate budgets, and reconcentrated later, and the timing of this maximum varies systematically with architecture and reasoning effort. An exact decomposition separates cohort-wide common-mode mass from residual spectral dimensionality: common-mode reallocation accounts for about two thirds of the trajectory, while the residual spectrum contributes about one quarter and retains substantial variation beyond the common mode. The decomposition further localizes behavior: among non-unanimous cohorts, increases in common-mode concentration strongly predict same-answer recoverability, and higher reasoning effort delays the maximum by 2.59 octaves (doublings of the token budget) and consistently expands the high-rank period across all four tested architectures, locating the effort effect in timing and duration rather than peak amplitude. Correctness comparisons separate structural monitoring from answer selection, positioning routing effective rank as a decomposable, label-free diagnostic of cohort organization - a principled spectral lens on how MoE reasoning cohorts differentiate and reconcentrate over inference time.

Related papers

• DiffIE: Diffusion-based Open Information Extraction [4.37919186746582]
  We introduce DIFFIE, which treats conditional discrete diffusion as the extraction mechanism itself.<n>DIFFIE achieves the new state of the art in CaRB, and outperforms the strongest rule-based system (ClausIE) in BenchIE.
  arXiv  Detail & Related papers  (2026-09-02T09:01:42Z)
• Beyond Random Partitioning: Unsupervised Spatio-Temporal Stratification for Cohort Balancing in Longitudinal Medical Imaging [0.9530026127073782]
  We formalize an unsupervised-temporal cohort-balancing standard operating procedure (SOP) that combines elbow-cluster K-means clustering over a standardized, six-dimensional joint intensity-temporal feature space.<n>On a longitudinal, contrast-enhanced $T1$-izable brain MRI cohort, the protocol reduces the maximum cross-subset intensity bias from 34.1% under conventional random shuffling to under 2.1%.
  arXiv  Detail & Related papers  (2026-07-29T09:33:19Z)
• Invariant Gradient Alignment for Robust Reasoning Distillation [9.39195684989942]
  Large language models (LLMs) fail on out-of-distribution (OOD) inputs whose semantic surface differs from training data, even when the logical structure is identical.<n>This undermines knowledge distillation pipelines that transfer chain-of-thought reasoning to smaller students.<n>We introduce Invariant Gradient Alignment (IGA), a training framework that aligns gradient updates across semantically diverse but logically isomorphic examples.
  arXiv  Detail & Related papers  (2026-06-03T15:48:52Z)
• AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE [51.994752158953084]
  AnchorMoE is an interpretable-by-construction classification framework.<n>It encodes multi-view representations of local patches and routes them to specialized experts.<n>Experiments on real-world and synthetic benchmarks demonstrate that AnchorMoE achieves highly competitive classification performance.
  arXiv  Detail & Related papers  (2026-06-02T13:30:54Z)
• Contrast to Detect: Dynamic Graph Contrastive Regularization for Unsupervised Anomaly Detection in Multivariate Time Series [24.192601578775186]
  We propose ContrastAD, an unsupervised framework that turns structural evolution into a learning signal rather than suppressing it.<n>Across five real-world benchmarks, ContrastAD attains the highest mean F1 and the highest AUC on three datasets.
  arXiv  Detail & Related papers  (2026-05-22T15:18:53Z)
• CaC: Advancing Video Reward Models via Hierarchical Spatiotemporal Concentrating [49.94171749437024]
  We propose Concentrate and Concentrate (CaC), a coarse-to-fine anomaly reward model based on Vision-Language Models.<n>We construct the first largescale generated video anomaly dataset with per-frame bounding-box annotations, temporal anomaly windows, and fine-grained attribution labels.<n>In experiments, CaC can stably concentrate on subtle anomalies, achieving a 25.7% accuracy improvement on fine-grained anomaly benchmarks.
  arXiv  Detail & Related papers  (2026-05-12T08:08:33Z)
• Continuous-Time Distribution Matching for Few-Step Diffusion Distillation [57.28746398500951]
  We introduce Continuous-Time Distribution Matching (CDM), migrating the DMD framework from discrete anchoring to continuous optimization.<n>CDM achieves this through two continuous-time designs. First, we replace the fixed discrete schedule with a dynamic continuous schedule of random length.<n>Second, we propose a continuous-time alignment objective that performs active off-trajectory matching on latents extrapolated via the student's velocity field.
  arXiv  Detail & Related papers  (2026-05-07T14:56:39Z)
• Loop-Extrusion Linkage: Spectral Ordering and Interval-Based Structure Discovery for Continuous Optimization [0.0]
  We introduce the Loop-Extrusion (LEL) operator, a structure-learning wrapper that combines adaptive variable-inter estimation, spectral seriation via the Fiedler vector, and adaptive interval-based subspace search.<n>We evaluate LEL on six diagnostic functions at d=96 designed to modulated specific structural hypotheses.<n>Results are assessed via the Wilcoxon signed-rank test with Holm-Bonferroni correction and Vargha-Delaney A12 effect sizes.
  arXiv  Detail & Related papers  (2026-04-05T21:26:21Z)
• On the Role of Reasoning Patterns in the Generalization Discrepancy of Long Chain-of-Thought Supervised Fine-Tuning [63.41902113656453]
  Supervised Fine-Tuning (SFT) on long Chain-of-Thought (CoT) trajectories has become a pivotal phase in building large reasoning models.<n>We conduct a comparative study using two sources of verified CoT trajectories generated by two competing models.<n>We find that SFT on textttDeepSeek-R1-0528 data achieves remarkably lower training loss, yet exhibits significantly worse generalization performance.
  arXiv  Detail & Related papers  (2026-04-02T07:00:54Z)
• A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization [0.0]
  Grokking the delayed transition from memorization to generalization in neural networks remains poorly understood.<n>We present a controlled study that systematically disentangles factors on modular addition (mod 97)<n>Our central finding is that grokking dynamics are not primarily determined by architecture, but by interactions between stability and regularization.
  arXiv  Detail & Related papers  (2026-03-26T04:16:01Z)
• WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training [64.0932926819307]
  We present Warmup-Stable and Merge (WSM), a framework that establishes a formal connection between learning rate decay and model merging.<n>WSM provides a unified theoretical foundation for emulating various decay strategies.<n>Our framework consistently outperforms the widely-adopted Warmup-Stable-Decay (WSD) approach across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-07-23T16:02:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.