From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models

• URL: http://arxiv.org/abs/2606.10703v2
• Date: Tue, 16 Jun 2026 11:15:27 GMT
• Title: From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models
• Abstract summary: We investigate the use of routing statistics in Mixture-of-Experts (MoE) pruning.<n>We find no observational metric predicts causal expert importance in any model.<n>Our results provide an explicit counterexample to the common inferential step from population-level observational summaries to token-level interventional claims about expert importance.
• Score: 0.2230291569252836
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Interpretability methods routinely use population-level summary statistics over observed model behaviour to license claims about the effects of targeted interventions on specific computations; in Pearl's terms, they treat rung-1 associational evidence as if it supported rung-2 interventional conclusions, a move whose validity is rarely tested. We examine one concrete instance: the use of routing statistics in Mixture-of-Experts (MoE) pruning, where utilization rates, activation norms, and routing weight distributions are treated as predictors of which experts can be removed without functional cost. A token-level interventional audit across three high-redundancy MoE architectures (OLMoE-1B-7B-0924, Qwen1.5-MoE-A2.7B, DeepSeek-V2-Lite) finds no observational metric predicts causal expert importance in any model: across all 60 metric-layer combinations effect sizes stay below Cohen's $d = 0.23$, and no metric is reliably positive under our corrected, dual-test criterion. A per-token routing weight control, run with identical $n$, rules out insufficient power, recovering a signal whose CI excludes zero at OLMoE's final MoE layer ($d = +0.231$, 95\% CI $[+0.09, +0.37]$, $p = 0.0013$). Existing pruning methods succeed in this regime not by identifying dispensable experts but because early-layer redundancy renders most selection criteria interchangeable. Our results provide an explicit counterexample to the common inferential step from population-level observational summaries to token-level interventional claims about expert importance, and illustrate how interventional audits can calibrate the evidential standards for interpretability claims.

Related papers

• Post-Hoc Trajectory-Risk Certification for Modular LLM-Based Security Agents [1.4915002678801434]
  We show that Bonferroni allocation is distribution-free but conservative under correlated errors.<n> coarse-to-fine label selection can create near-perfect measured correlation without learned dependence.<n>A modular certificate using per-stage certificates and a pairwise overlap bound yields a positive average gain of 0.6%.
  arXiv  Detail & Related papers  (2026-08-04T23:48:36Z)
• Modeling Normal Is All You Need: Joint Latent Clustering for Anomaly Detection in Multimodal Cyber-Physical Systems [47.027290803102666]
  Normal behaviour is the union of many imbalanced, curved, thin-fringed operating regimes.<n>We model the normal law with a jointly learned latent representation plus explicit Gaussian-mixture mode clustering.<n>We evaluate under a deliberately fair protocol: raw point-wise metrics with no point adjustment, a trivial-detector difficulty split, prevalence-matched F1, and train-normal-only calibration.
  arXiv  Detail & Related papers  (2026-07-07T10:10:00Z)
• Nous: An Attempt to Extract and Inject the Cognition Behind Prediction-Market Behavior [0.0]
  We ask whether human cognitive diversity can be recovered from behavior and transferred to LLM agents.<n>Nous extracts a structured eight-dimension behavioral profile from real Polymarket trading activity and injects it into agents through prompts.<n>We position Nous as measuring the cognitive-monoculture problem and the limits of a prompt-level remedy.
  arXiv  Detail & Related papers  (2026-06-11T08:18:25Z)
• Causal Inference with Categorical Unobserved Confounder via Mixture Learning [20.542714966688656]
  We show that causal effects are identifiable when the unobserved confounder is categorical under suitable conditions.<n>We propose an estimation procedure based on tensor decomposition, which allows consistent recovery of the latent structure.
  arXiv  Detail & Related papers  (2026-05-18T18:28:16Z)
• Active Multiple-Prediction-Powered Inference [2.1710524042938073]
  Post-deployment monitoring of healthcare requires statistically valid, label-efficient methods, but gold-standard labels from clinician chart review are expensive.<n>We propose Prediction Active-Prediction-Powered Inference (AM-PPI), which routes each instance to a cost-appropriate predictor, samples gold-standard labels in proportion, and repredicts predictions to a single deployment-time.
  arXiv  Detail & Related papers  (2026-05-08T19:48:57Z)
• How Independent are Large Language Models? A Statistical Framework for Auditing Behavioral Entanglement and Reweighting Verifier Ensembles [46.63622714488747]
  Shared pretraining data, distillation, and alignment pipelines can induce hidden behavioral dependencies, latent entanglement.<n>In practice, this manifests as correlated reasoning patterns and synchronized failures.<n>We develop a statistical framework for auditing behavioral entanglement among black-box language models.
  arXiv  Detail & Related papers  (2026-04-08T23:32:06Z)
• SELFDOUBT: Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratio [0.038379177968040606]
  We propose SELFDOUBT, a single-pass uncertainty framework for reasoning language models.<n>Our key signal, the Hedge-to-Verify Ratio (HVR), detects whether a reasoning trace contains uncertainty markers and, if so, whether they are offset by explicit selfchecking behavior.<n> SELFDOUBT operates on a single observed reasoning trajectory, making it suitable for latency- and cost-constrained deployment over any proprietary API.
  arXiv  Detail & Related papers  (2026-04-07T19:19:29Z)
• Beyond Completion: Probing Cumulative State Tracking to Predict LLM Agent Performance [9.771590610969918]
  We introduce Working Memory Fidelity-Active Manipulation (WMF-AM)<n>We evaluate it on 20 open-weight models (0.5B-35B, 13 families) against a released deterministic 10-task agent battery.
  arXiv  Detail & Related papers  (2026-03-28T17:25:11Z)
• Scalable Contrastive Causal Discovery under Unknown Soft Interventions [3.165716101116899]
  We propose a scalable causal discovery model for paired observational and interventional settings with shared underlying causal structure and unknown soft interventions.<n> Experiments on synthetic data demonstrate improved causal structure recovery, generalization to unseen graphs with held-out causal mechanisms, and scalability to larger graphs.
  arXiv  Detail & Related papers  (2026-03-03T18:16:16Z)
• STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction [78.0692157478247]
  We propose STAR, a framework that bridges data-driven STatistical expectations with knowledge-driven Agentic Reasoning.<n>We show that STAR consistently outperforms all baselines on both score-based and rank-based metrics.
  arXiv  Detail & Related papers  (2026-02-12T16:30:07Z)
• Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
  We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
  arXiv  Detail & Related papers  (2026-02-10T21:08:53Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• Flexible Nonparametric Inference for Causal Effects under the Front-Door Model [2.6900047294457683]
  We develop novel one-step and targeted minimum loss-based estimators for both the average treatment effect and the average treatment effect on the treated under front-door assumptions.<n>Our estimators are built on multiple parameterizations of the observed data distribution, including approaches that avoid mediator density entirely.<n>We show how these constraints can be leveraged to improve the efficiency of causal effect estimators.
  arXiv  Detail & Related papers  (2023-12-15T22:04:53Z)
• Hierarchical Semi-Supervised Contrastive Learning for Contamination-Resistant Anomaly Detection [81.07346419422605]
  Anomaly detection aims at identifying deviant samples from the normal data distribution. Contrastive learning has provided a successful way to sample representation that enables effective discrimination on anomalies. We propose a novel hierarchical semi-supervised contrastive learning framework, for contamination-resistant anomaly detection.
  arXiv  Detail & Related papers  (2022-07-24T18:49:26Z)
• Counterfactual Maximum Likelihood Estimation for Training Deep Networks [83.44219640437657]
  Deep learning models are prone to learning spurious correlations that should not be learned as predictive clues. We propose a causality-based training framework to reduce the spurious correlations caused by observable confounders. We conduct experiments on two real-world tasks: Natural Language Inference (NLI) and Image Captioning.
  arXiv  Detail & Related papers  (2021-06-07T17:47:16Z)
• Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap [140.98628848491146]
  We introduce deconfounding scores, which induce better overlap without biasing the target of estimation. We show that deconfounding scores satisfy a zero-covariance condition that is identifiable in observed data. In particular, we show that this technique could be an attractive alternative to standard regularizations.
  arXiv  Detail & Related papers  (2021-04-12T18:50:11Z)
• Foreseeing the Benefits of Incidental Supervision [83.08441990812636]
  This paper studies whether we can, in a single framework, quantify the benefits of various types of incidental signals for a given target task without going through experiments. We propose a unified PAC-Bayesian motivated informativeness measure, PABI, that characterizes the uncertainty reduction provided by incidental supervision signals.
  arXiv  Detail & Related papers  (2020-06-09T20:59:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.