Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation

• URL: http://arxiv.org/abs/2608.15787v1
• Date: Sun, 16 Aug 2026 15:07:17 GMT
• Title: Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation
• Abstract summary: Two Mixture-of-Experts (MoE) forward passes can share every weight yet route the same token through different experts.<n>We study this mismatch in its single-step form, with frozen weights rather than as a proxy for a full training trajectory.
• Score: 7.901904409617788
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Two Mixture-of-Experts (MoE) forward passes can share every weight yet route the same token through different experts. This creates a possible blind spot in same-weight self-distillation, where a demonstration-conditioned teacher supervises a query-only student. We study this mismatch in its single-step form, with frozen weights rather than as a proxy for a full training trajectory. An exact blockwise decomposition separates a routing term, which changes gates at fixed content, from a dense-like content term. Across seven open-weight checkpoints and two domains, the routing term spans only $1.6\times$ as a fraction of block output, while its residual-stream exposure spans $3.2\times$. Exposure is ordered by the routed block's share of the residual. Scaling the always-on backbone in two confirmatory models moves exposure monotonically; common-mode controls support a mass-and-coherence mechanism rather than denominator dilution alone. Preregistered PubMedQA patches on three models show that the full routing term moves outputs by less than half the natural context effect and is largely reproduced by matched-norm noise, whereas the content term is strongly direction-specific. Scale and merged-expert probes show that the narrow block-level range is not universal, although exposure remains small at the tested boundaries. Router movement alone is therefore not evidence of behavioral influence: measure exposure first, and use a behavioral intervention when the decision matters.

Related papers

• THESIS-MoE: Trainable Hierarchical Extraction and SteerIng of Sycophancy in Mixture-of-Experts [0.300988853836121]
  Sycophancy, the tendency of a language model to change its answer to match a user's stated belief, is a common alignment failure.<n>We introduce a shared contrastive signal, built from matched prompts with and without a stated belief, that identifies where sycophancy lives.<n>Our results demonstrate that sycophancy resides in identifiable computational subcircuits and can be selectively steered.
  arXiv  Detail & Related papers  (2026-08-16T11:30:26Z)
• Orientation, not magnitude: the causal structure of task-vector interference in merged language models [0.0]
  We show that task arithmetic works until it doesn't, and the field diagnoses why with magnitudes.<n>An exact decomposition of the layerwise flux shows it is dominated by amplifying transport of the existing cross-term.<n>The striking +-15% "universality" of naive bfloat16 generation turns out to be quantization roughness.
  arXiv  Detail & Related papers  (2026-08-12T08:40:24Z)
• Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation [49.8459545291965]
  Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.<n>We study this dependency by decomposing sparsely upcycled feed-forward experts into key-value channels.<n>We instantiate it in UniF-MoE, a unified framework for token-adaptive MoE computation.
  arXiv  Detail & Related papers  (2026-08-11T02:40:58Z)
• Dual-Branch Cross-Projection Debiasing through Diffusion-based Disentanglement [66.67463557274358]
  Foundation models trained on biased datasets often rely on spurious correlations between target labels and non-causal attributes.<n>We introduce Confidence-guided Concept Mining (CBCM) to identify reliable spurious attributes without attribute annotations.<n>We propose Dual-branch Cross-projection Debiasing (DCD), a prompt-tuning framework that separates target and spurious representations into two branches.
  arXiv  Detail & Related papers  (2026-06-23T05:28:47Z)
• GEMS: Geometric Constraints Enable Multi-Semantic Superposition in LLMs [5.018957704056544]
  Activation steering controls model behavior by modifying intermediate hidden states at inference time without retraining.<n>We show that when multiple semantic directions are superposed without constraints, the model collapses.<n>We propose GEMS, a training-free method that maps each source to a corresponding geometric constraint.
  arXiv  Detail & Related papers  (2026-06-18T08:43:33Z)
• When Does Routing Become Interpretable? Causal Probes on Block Attention Residuals [0.0]
  Block Attention Residuals (Block AttnRes) replace fixed additive residuals with a learned softmax over earlier depth-source representations.<n>We ask whether such exposure suffices for mechanistic interpretation.
  arXiv  Detail & Related papers  (2026-06-11T10:37:04Z)
• Boundary Mass and the Soft-to-Hard Limit in Mixture-of-Experts [0.0]
  We study the singularity at the population level for squared-loss MoE regression.<n>The central object is the emphboundary mass, namely the probability that the top two router scores are separated by only a small margin.<n>We show that the zero-temperature limit is controlled by a thin geometric layer around routing interfaces, not by the full input space.
  arXiv  Detail & Related papers  (2026-05-04T01:07:21Z)
• Primary-Fine Decoupling for Action Generation in Robotic Imitation [91.2899765310853]
  Multi-modal distribution in robotic manipulation action sequences poses critical challenges for imitation learning.<n>We propose Primary-Fine Decoupling for Action Generation (PF-DAG), a two-stage framework that decouples coarse action consistency from fine-grained variations.<n>PF-DAG outperforms state-of-the-art baselines across 56 tasks from Adroit, DexArt, and MetaWorld benchmarks.
  arXiv  Detail & Related papers  (2026-02-25T08:36:45Z)
• Counterfactual Realizability [52.85109506684737]
  We introduce a formal definition of realizability, the ability to draw samples from a distribution, and then develop a complete algorithm to determine whether an arbitrary counterfactual distribution is realizable.<n>We illustrate the implications of this new framework for counterfactual data collection using motivating examples from causal fairness and causal reinforcement learning.
  arXiv  Detail & Related papers  (2025-03-14T20:54:27Z)
• Learning a Fast Mixing Exogenous Block MDP using a Single Trajectory [87.62730694973696]
  STEEL is the first provably sample-efficient algorithm for learning the controllable dynamics of an Exogenous Block Markov Decision Process from a single trajectory.<n>We prove that STEEL is correct and sample-efficient, and demonstrate STEEL on two toy problems.
  arXiv  Detail & Related papers  (2024-10-03T21:57:21Z)
• Clustered Switchback Designs for Experimentation Under Spatio-temporal Interference [44.644520116360106]
  We estimate the global average treatment effect (GATE), the difference between average outcomes having exposed all units at all times to treatment or to control.<n>We propose a clustered switchback design, where units are grouped into clusters and time steps are grouped into blocks.<n>We show that for graphs that admit good clustering, a truncated Horvitz-Thompson estimator achieves a $tilde O(1/NT)$ mean squared error (MSE)<n>Our results simultaneously generalize the results from citethu2022switchback,ugander2013graph and citetleung2022rate
  arXiv  Detail & Related papers  (2023-12-25T01:00:58Z)
• Evolution of many-body systems under ancilla quantum measurements [58.720142291102135]
  We study the concept of implementing quantum measurements by coupling a many-body lattice system to an ancillary degree of freedom. We find evidence of a disentangling-entangling measurement-induced transition as was previously observed in more abstract models.
  arXiv  Detail & Related papers  (2023-03-13T13:06:40Z)
• StableMoE: Stable Routing Strategy for Mixture of Experts [109.0602120199226]
  Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead. We propose StableMoE with two training stages to address the routing fluctuation problem. Results show that StableMoE outperforms existing MoE methods in terms of both convergence speed and performance.
  arXiv  Detail & Related papers  (2022-04-18T16:48:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.