Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations

• URL: http://arxiv.org/abs/2604.14246v1
• Date: Wed, 15 Apr 2026 06:21:47 GMT
• Title: Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations
• Abstract summary: We propose Counterfactual Routing (CoR), a training-free inference framework designed to awaken experts.<n>CoR integrates layer-wise perturbation analysis with the Counterfactual Expert Impact (CEI) metric to dynamically shift computational resources.<n>Experiments on TruthfulQA, FACTOR, and TriviaQA demonstrate that CoR improves factual accuracy by 3.1% on average without increasing the inference budget.
• Score: 47.02065120237642
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse Mixture-of-Experts (MoE) models have achieved remarkable scalability, yet they remain vulnerable to hallucinations, particularly when processing long-tail knowledge. We identify that this fragility stems from static Top-$k$ routing: routers tend to favor high-frequency patterns over rare factual associations. Consequently, ``specialist experts'' possessing critical long-tail knowledge are often assigned low gating scores and remain ``dormant'' -- under-prioritized for specific tokens despite their proven causal importance on other inputs. To address this, we propose Counterfactual Routing (CoR), a training-free inference framework designed to awaken these dormant experts. CoR integrates layer-wise perturbation analysis with the Counterfactual Expert Impact (CEI) metric to dynamically shift computational resources from syntax-dominant to knowledge-intensive layers while maintaining a constant total activation count, effectively retrieving causally decisive experts via virtual ablation. Extensive experiments on TruthfulQA, FACTOR, and TriviaQA demonstrate that CoR improves factual accuracy by 3.1\% on average without increasing the inference budget, establishing a superior Pareto frontier compared to static scaling strategies.

Related papers

• AnchorMoE: Interpretable Time Series Classification via Anchor-Routed MoE [51.994752158953084]
  AnchorMoE is an interpretable-by-construction classification framework.<n>It encodes multi-view representations of local patches and routes them to specialized experts.<n>Experiments on real-world and synthetic benchmarks demonstrate that AnchorMoE achieves highly competitive classification performance.
  arXiv  Detail & Related papers  (2026-06-02T13:30:54Z)
• Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning [4.874690972824275]
  We propose an auxiliary-loss-free MoE SFT framework that combines bias-driven sparsification with always-active gated condenser experts.<n>Our approach outperforms state-of-the-art SFT baselines such as DenseMixer and ESFT, achieving average gain of 2.5%+ on both mathematical reasoning and commonsenseQA benchmarks.
  arXiv  Detail & Related papers  (2026-04-24T21:48:20Z)
• DK-Root: A Joint Data-and-Knowledge-Driven Framework for Root Cause Analysis of QoE Degradations in Mobile Networks [17.511985643225618]
  We present DK-Root, a joint data-and-knowledge-driven framework for root-cause analysis.<n>It unifies scalable weak supervision with precise expert guidance for robust root-cause analysis.<n>Experiments on a real-world, operator-grade dataset demonstrate state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2025-11-13T09:32:01Z)
• Beyond Redundancy: Diverse and Specialized Multi-Expert Sparse Autoencoder [59.89996751196727]
  Sparse autoencoders (SAEs) have emerged as a powerful tool for interpreting large language models.<n>SAEs' hidden layers have high dimensionality to satisfy sparsity constraints, resulting in prohibitive training and inference costs.<n>Recent Mixture of Experts (MoE) approaches attempt to address this by SAEs into narrower expert networks with gated activation.<n>We propose two key innovations: (1) Multiple Expert Activation that simultaneously engages semantically weighted expert subsets to encourage specialization, and (2) Feature Scaling that enhances diversity through adaptive high-frequency scaling.
  arXiv  Detail & Related papers  (2025-11-07T22:19:34Z)
• Going Beyond Expert Performance via Deep Implicit Imitation Reinforcement Learning [3.691573844585973]
  We introduce a deep implicit imitation reinforcement learning framework that combines deep reinforcement learning with implicit imitation learning from observation-only datasets.<n>Our main algorithm, Deep Implicit Q-Network (DIIQN), employs an action inference mechanism that reconstructs expert actions through online exploration.<n>We further extend our framework with a Heterogeneous Actions DIIQN (HA-DIIQN) algorithm to tackle scenarios where expert and agent possess different action sets.
  arXiv  Detail & Related papers  (2025-11-05T16:33:39Z)
• LEAF: A Robust Expert-Based Framework for Few-Shot Continual Event Detection [7.094483187879095]
  LEAF is a novel and robust expert-based framework for Continual Event Detection.<n>It incorporates a specialized mixture of experts architecture into the base model, where each expert is parameterized with low-rank adaptation (LoRA) matrices.<n>A semantic-aware expert selection mechanism dynamically routes instances to the most relevant experts, enabling expert specialization and reducing knowledge interference.
  arXiv  Detail & Related papers  (2025-09-29T10:00:25Z)
• One-Prompt Strikes Back: Sparse Mixture of Experts for Prompt-based Continual Learning [52.966712416640085]
  We propose SMoPE, a novel framework that integrates the benefits of both task-specific and shared prompt strategies.<n>SMoPE consistently outperforms task-specific prompt methods and achieves performance competitive with state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-09-29T08:54:58Z)
• ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation [91.20492150248106]
  We investigate the internal mechanisms behind unfaithful generation and identify a subset of mid-to-deep feed-forward networks (FFNs) that are disproportionately activated in such cases.<n>We propose Parametric Knowledge Muting through FFN Suppression (ParamMute), a framework that improves contextual faithfulness by suppressing the activation of unfaithfulness-associated FFNs.<n> Experimental results show that ParamMute significantly enhances faithfulness across both CoFaithfulQA and the established ConFiQA benchmark, achieving substantial reductions in reliance on parametric memory.
  arXiv  Detail & Related papers  (2025-02-21T15:50:41Z)
• Mixture of Tunable Experts - Behavior Modification of DeepSeek-R1 at Inference Time [1.1655046053160683]
  We present a method that extends the Mixture-of-Experts architecture of Large Language Models (LLMs)<n>MoTE enables meaningful and focused behavior changes in LLMs on-the-fly during inference time.
  arXiv  Detail & Related papers  (2025-02-16T12:24:39Z)
• Ladder-of-Thought: Using Knowledge as Steps to Elevate Stance Detection [73.31406286956535]
  We introduce the Ladder-of-Thought (LoT) for the stance detection task. LoT directs the small LMs to assimilate high-quality external knowledge, refining the intermediate rationales produced. Our empirical evaluations underscore LoT's efficacy, marking a 16% improvement over GPT-3.5 and a 10% enhancement compared to GPT-3.5 with CoT on stance detection task.
  arXiv  Detail & Related papers  (2023-08-31T14:31:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.