How Modular Is a Frontier Mixture-of-Experts? A Pre-registered Causal Test in Which Apparent Expert Modularity Mostly Dissolves

• URL: http://arxiv.org/abs/2606.25092v1
• Date: Tue, 23 Jun 2026 18:55:41 GMT
• Title: How Modular Is a Frontier Mixture-of-Experts? A Pre-registered Causal Test in Which Apparent Expert Modularity Mostly Dissolves
• Abstract summary: We build a routing-mass atlas, pre-register six family-to-axis hypotheses, and ablate each family at inference time against a size-matched random-expert null.<n>We test the same families under four metrics and a held-out, independent-corpus run with confidence bootstrap intervals.<n>Of six pre-registered families, only one, the Arabic-language family, is a clean selective module that survives an independent corpus and a conservative statistical bar.
• Score: 0.08599681538174887
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse Mixture-of-Experts (MoE) models route each token to a few of many experts, inviting the hypothesis that experts form functional modules tied to capabilities or languages. We test this causally on Command A+, a frontier open-weights MoE (218B total / 25B active; 128 experts, 8 active, +1 shared). We build a routing-mass atlas, pre-register six family-to-axis hypotheses before any intervention, and ablate each family at inference time against a size-matched random-expert null, measuring whether it selectively breaks its own axis (worst off-target effect at most one third of on-target). Crucially, we test the same families under four metrics and a held-out, independent-corpus run with bootstrap confidence intervals. Our finding is cautionary: robust functional modularity is rare and measurement-dependent. Of six pre-registered families, only one, the Arabic-language family, is a clean selective module that survives an independent corpus and a conservative statistical bar (1/6; a more permissive pre-registered point rule admits 3/6, but that count is threshold-sensitive). Every other family has a real causal effect yet fails selectivity, and its apparent modularity flips with the measurement: with the corpus, the metric, and the statistical bar. A positive control on Qwen3-30B-A3B recovers its published disjoint structure, confirming the method detects modularity when present. The verdict reproduces on the un-quantized BF16 model, ruling out a 4-bit quantization artifact. We conclude that ablation-based modularity verdicts are not safe unless the corpus, metric, and statistical bar are controlled. We release the atlas and ablation data.

Related papers

• When Do Internal Probes Beat Reading the Answer? Miscalibrated Readouts and Behavior-Concealed Knowledge in Language Models [0.0]
  A language model, asked to verify 1,200 logical conclusions, answers YES every time.<n>linear probes on its hidden states read the correct verdict at 0.96 AUC.<n>We ask where the verdict is lost, and find the dominant failure is a single scalar.<n>The diagnosis generalizes: across 90 semantic-label configurations of a five-model, three-family factorial, behavioral accuracy collapses onto a single function of threshold offset.
  arXiv  Detail & Related papers  (2026-09-04T00:26:45Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation [0.0]
  We document a failure class in frontier large language models.<n>The failure reproduces at first observation, but its empirical surface is unstable.<n>For regulated, frontier-model accuracy is a moving compliance boundary that shifts without notice.
  arXiv  Detail & Related papers  (2026-07-25T22:40:02Z)
• Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families [51.56484100374058]
  Abliteration - deleting a model's refusal direction from its weights - is the standard recipe behind popular "uncensored" open-weight models.<n>As a disposition probe we use 21,600 decisions under uncertainty, replayed through a frozen pipeline so the decision-layer model is the only variable.<n>Three effects replicate across both families (weeks-clustered CIs excluding zero)<n>A fourth effect reverses sign: the same operation makes Gemma-abliterated less confident and Qwen-abliterated more (family CIs non-overlapping)
  arXiv  Detail & Related papers  (2026-07-19T22:27:00Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• When Is Emergent Consensus Real? A Measured Coupling Gain and a Validity Diagnostic for LLM Agent Societies [1.2763567932588586]
  LLM "agent societies" are studied via demonstrations of emergent consensus or polarization.<n>No test of whether an outcome is a genuine social dynamic or a model artifact.<n>We introduce the coupling gain+, measured per-agent by counterfactually perturbing a neighbour's stated opinion.
  arXiv  Detail & Related papers  (2026-06-20T19:41:01Z)
• Traits Run Deeper: Trait-Specific Asymmetric Fusion for Personality Assessment [53.50824432699408]
  Traits Run Deeper is a novel personality assessment framework.<n>It consists of three components: the Multimodal Foundation Representation (MFR), the Trait-Specific Modality Fusion (TSMF) and the Distribution-Calibrated Personality Regression (DCPR)
  arXiv  Detail & Related papers  (2026-06-09T06:38:36Z)
• Measuring and Improving Behavioral Consistency in Large Language Models through Fact-Heuristic-Emotion State Enforcement [4.226475360842309]
  Large language models (LLMs) can give different answers to the same decision problem across runs, and reverse a decision when their own prior answer returns as context.<n>We ask whether this instability can be measured and partially reduced without changing model weights.<n>We test the Cognitive Kernel Model (CKM), a prompt-level state-enforcement layer.
  arXiv  Detail & Related papers  (2026-06-05T12:36:36Z)
• TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models [47.0940918286083]
  We introduce the Tangent Upper Bound on Evidence (TUBE), a variational upper bound on log-likelihood that admits an unbiased Monte Carlo estimator.<n>Our TUBE extends across latent-variable models, including masked diffusion models (MDMs), any-order ARMs (AO-ARMs) and block variants of both.
  arXiv  Detail & Related papers  (2026-05-22T23:48:25Z)
• Three Regimes of Context-Parametric Conflict: A Predictive Framework and Empirical Validation [0.0]
  We study how large language models handle conflict between their training knowledge and a contradicting document.<n>We propose a three-regime framework: Regime 1 (single-source updating), Regime 2 (competitive integration), and Regime 3 (task-appropriate selection)<n>We validate the framework across Claude Sonnet 4.6, GPT-5.5, Gemini 2.5 Flash, Llama 4 Maverick, and DeepSeek V3 using 9,970 API calls in three experimental phases.
  arXiv  Detail & Related papers  (2026-05-12T06:00:48Z)
• The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive [0.0]
  CPU-only scoring primitive runs at 2.6 microseconds per token.<n> token rank-frequency distributions converge to the same two- parameter Mandelbrot ranking distribution.<n>We derive a single-pass scoring primitive that composes with model log probabilities when available and degrades to a rank-only mode usable on closed APIs.
  arXiv  Detail & Related papers  (2026-04-28T13:35:31Z)
• Statistics, Not Scale: Modular Medical Dialogue with Bayesian Belief Engine [51.722324399751294]
  Large language models are increasingly deployed as autonomous diagnostic agents, yet they conflate two fundamentally different capabilities.<n>We introduce BMBE, a modular diagnostic dialogue framework that enforces a strict separation between language and reasoning.
  arXiv  Detail & Related papers  (2026-04-21T21:59:57Z)
• How Independent are Large Language Models? A Statistical Framework for Auditing Behavioral Entanglement and Reweighting Verifier Ensembles [46.63622714488747]
  Shared pretraining data, distillation, and alignment pipelines can induce hidden behavioral dependencies, latent entanglement.<n>In practice, this manifests as correlated reasoning patterns and synchronized failures.<n>We develop a statistical framework for auditing behavioral entanglement among black-box language models.
  arXiv  Detail & Related papers  (2026-04-08T23:32:06Z)
• Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models [55.94503936470247]
  Large-scale AI evaluation increasingly relies on aggregating binary judgments from $K$ annotators, including judges.<n>Most classical methods assume annotators are conditionally independent given the true label $Yin0,1$, an assumption often violated by LLM judges.<n>We study label aggregation through a hierarchy of dependence-aware models based on Ising graphical models and latent factors.
  arXiv  Detail & Related papers  (2026-01-29T21:26:50Z)
• Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles [2.957223821964636]
  BatchEnsemble aims to deliver ensemble-like uncertainty (EU) EU at far lower parameter and memory cost.<n>We show that BatchEnsemble not only underperforms Deep Ensembles but closely tracks a single model baseline.
  arXiv  Detail & Related papers  (2026-01-23T17:50:50Z)
• Intention Collapse: Intention-Level Metrics for Reasoning in Language Models [0.0]
  We call this process intention collapse: a many-to-one projection from a high dimensional intention space I into an external language space L.<n>We define three model agnostic intention metrics (intention entropy, effective dimensionality dimeff, and latent knowledge recoverability Recov)<n>Using a 4 bit Mistral 7B model on 200 GSM8K problems, we compare a direct answer baseline, a chain of thought (CoT) regime, and a babble control.
  arXiv  Detail & Related papers  (2026-01-03T00:19:53Z)
• Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learning [48.14470449860784]
  Test-time reinforcement learning (TTRL) offers a label-free paradigm for adapting models using only synthetic signals at inference.<n>We introduce Self-Harmony, a framework built on a simple intuition.
  arXiv  Detail & Related papers  (2025-11-03T03:34:34Z)
• Independence Tests for Language Models [47.0749292650885]
  Given the weights of two models, can we test whether they were trained independently?<n>We consider two settings: constrained and unconstrained.<n>We propose a new test which matches hidden activations between two models, and which is robust to adversarial transformations and to changes in model architecture.
  arXiv  Detail & Related papers  (2025-02-17T20:01:08Z)
• Model-free Methods for Event History Analysis and Efficient Adjustment (PhD Thesis) [55.2480439325792]
  This thesis is a series of independent contributions to statistics unified by a model-free perspective.<n>The first chapter elaborates on how a model-free perspective can be used to formulate flexible methods that leverage prediction techniques from machine learning.<n>The second chapter studies the concept of local independence, which describes whether the evolution of one process is directly influenced by another.
  arXiv  Detail & Related papers  (2025-02-11T19:24:09Z)
• Confidence-Based Model Selection: When to Take Shortcuts for Subpopulation Shifts [119.22672589020394]
  We propose COnfidence-baSed MOdel Selection (CosMoS), where model confidence can effectively guide model selection. We evaluate CosMoS on four datasets with spurious correlations, each with multiple test sets with varying levels of data distribution shift.
  arXiv  Detail & Related papers  (2023-06-19T18:48:15Z)
• Distributed, partially collapsed MCMC for Bayesian Nonparametrics [68.5279360794418]
  We exploit the fact that completely random measures, which commonly used models like the Dirichlet process and the beta-Bernoulli process can be expressed as, are decomposable into independent sub-measures. We use this decomposition to partition the latent measure into a finite measure containing only instantiated components, and an infinite measure containing all other components. The resulting hybrid algorithm can be applied to allow scalable inference without sacrificing convergence guarantees.
  arXiv  Detail & Related papers  (2020-01-15T23:10:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.