Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE

• URL: http://arxiv.org/abs/2602.02443v1
• Date: Mon, 02 Feb 2026 18:39:33 GMT
• Title: Certain Head, Uncertain Tail: Expert-Sample for Test-Time Scaling in Fine-Grained MoE
• Abstract summary: Fine-grained MoE features hundreds of well-trained experts per layer and multi-expert activation per token.<n>We empirically characterize fine-grained MoE routing and uncover an informative pattern: router scores exhibit a certain head of high-confidence experts followed by an uncertain tail of low-confidence candidates.<n>We propose Expert-Sample, a training-free method that preserves high-confidence selections while injecting controlledity into the uncertain tail, enabling diverse generation without destabilizing outputs.
• Score: 27.8012190589404
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test-time scaling improves LLM performance by generating multiple candidate solutions, yet token-level sampling requires temperature tuning that trades off diversity against stability. Fine-grained MoE, featuring hundreds of well-trained experts per layer and multi-expert activation per token, offers an unexplored alternative through its rich routing space. We empirically characterize fine-grained MoE routing and uncover an informative pattern: router scores exhibit a certain head of high-confidence experts followed by an uncertain tail of low-confidence candidates. While single-run greedy accuracy remains stable when fewer experts are activated, multi-sample pass@n degrades significantly-suggesting that the certain head governs core reasoning capability while the uncertain tail correlates with reasoning diversity. Motivated by these findings, we propose Expert-Sample, a training-free method that preserves high-confidence selections while injecting controlled stochasticity into the uncertain tail, enabling diverse generation without destabilizing outputs. Evaluated on multiple fine-grained MoE models across math, knowledge reasoning, and code tasks, Expert-Sample consistently improves pass@n and verification-based accuracy. On Qwen3-30B-A3B-Instruct evaluated on GPQA-Diamond with 32 parallel samples, pass@32 rises from 85.4% to 91.9%, and accuracy improves from 59.1% to 62.6% with Best-of-N verification.

Related papers

• POOL: Propagated Uncertainty Over Lookalikes [17.512476090072028]
  textscPOOL (emphPropagated Uncertainty Over Lookalikes) is a cost-efficient framework that addresses this trade-off.<n>We instantiate this framework with textscHy@$p$, a hybrid estimator that combines verbal confidence with spectral answer diversity.<n>textscHy@5 achieves higher average AUROC than verbal confidence and textscVn@10 sampling while using half as many samples as textscVn@10.
  arXiv  Detail & Related papers  (2026-08-24T10:51:44Z)
• Learning What to Fail On: Failure-Mode Contextual Bandits for Adversarial Data Curation [14.914697247710622]
  We introduce a failure-aware adversarial retrieval-augmented framework for improving robustness in natural language understanding.<n>Our approach improves RoBERTa-base accuracy from 88.48% to 92.60% on SNLI, from 75.04% to 80.95% on ANLI, and from 54.67% to 71.99% on MultiNLI.<n>We demonstrate transfer to FEVER fact verification, achieving up to 79.86% FEVER score and 82.45% accuracy with RoBERTa-large.
  arXiv  Detail & Related papers  (2026-08-19T08:31:15Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• TEMPO: Scaling Test-time Training for Large Reasoning Models [87.61789183311856]
  Test-time training (TTT) adapts model parameters on unlabeled test instances during inference time.<n>We propose TEMPO, a TTT framework that interleaves policy refinement on unlabeled questions with periodic critic recalibration on a labeled dataset.
  arXiv  Detail & Related papers  (2026-04-21T10:01:04Z)
• SELFDOUBT: Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratio [0.038379177968040606]
  We propose SELFDOUBT, a single-pass uncertainty framework for reasoning language models.<n>Our key signal, the Hedge-to-Verify Ratio (HVR), detects whether a reasoning trace contains uncertainty markers and, if so, whether they are offset by explicit selfchecking behavior.<n> SELFDOUBT operates on a single observed reasoning trajectory, making it suitable for latency- and cost-constrained deployment over any proprietary API.
  arXiv  Detail & Related papers  (2026-04-07T19:19:29Z)
• Diffusion-MPC in Discrete Domains: Feasibility Constraints, Horizon Effects, and Critic Alignment: Case study with Tetris [0.0]
  We study diffusion-based model predictive control (Diffusion-MPC) in discrete domains using Tetris as a case study.<n>Our planner samples candidate placement sequences with a MaskGIT-style discrete denoiser and selects actions via reranking.<n>We analyze three key factors: (1) feasibility-constrained sampling via logit masking over valid placements, (2) reranking strategies using a score, a pretrained DQN critic, and a hybrid combination, and (3) compute scaling in candidate count and planning horizon.
  arXiv  Detail & Related papers  (2026-03-02T19:35:38Z)
• ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference [60.958331943869126]
  ODAR-Expert is an adaptive routing framework that optimize the accuracy-efficiency trade-off via principled resource allocation.<n>We show strong and consistent gains, including 98.2% accuracy on MATH and 54.8% on Humanity's Last Exam.
  arXiv  Detail & Related papers  (2026-02-27T05:22:01Z)
• Improving Sampling for Masked Diffusion Models via Information Gain [9.059619122219502]
  Masked Diffusion Models (MDMs) offer greater flexibility in decoding order than autoregressive models.<n> Existing samplers typically adopt greedys, prioritizing positions with the highest local certainty to decode at each step.<n>We propose the Info-Gain Sampler, a principled decoding framework that balances immediate uncertainty with information gain.
  arXiv  Detail & Related papers  (2026-02-20T12:26:03Z)
• LYNX: Learning Dynamic Exits for Confidence-Controlled Reasoning [15.597220136913258]
  LYNX is an online early-exit mechanism that turns a model's own hidden-state awareness into confidence-controlled stopping decisions.<n>We train and calibrate this probe once on a generic mathematical corpus and reuse it unchanged across benchmarks, decoding temperatures, and even non-mathematical tasks.
  arXiv  Detail & Related papers  (2025-12-05T00:04:42Z)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs [72.82403830490084]
  We argue that the decoding rule should be calibrated by correctness, not confidence alone.<n>We propose simple strategies that achieve this goal: Greedy-Threshold makes sampling greedy at very low confidence steps.<n>Together, our findings challenge prevailings about decoding under uncertainty and show gains across math and general reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-07T14:46:12Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• ORFuzz: Fuzzing the "Other Side" of LLM Safety -- Testing Over-Refusal [27.26251627767238]
  Large Language Models (LLMs) increasingly exhibit over-refusal - erroneously rejecting benign queries due to overly conservative safety measures.<n>This paper introduces the first evolutionary testing framework, ORFuzz, for the systematic detection and analysis of LLM over-refusals.
  arXiv  Detail & Related papers  (2025-08-15T05:03:26Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Adversarial Robustness Overestimation and Instability in TRADES [4.063518154926961]
  TRADES sometimes yields disproportionately high PGD validation accuracy compared to the AutoAttack testing accuracy in the multiclass classification task. This discrepancy highlights a significant overestimation of robustness for these instances, potentially linked to gradient masking.
  arXiv  Detail & Related papers  (2024-10-10T07:32:40Z)
• Certified Robust Accuracy of Neural Networks Are Bounded due to Bayes Errors [3.350980549219263]
  certified training improves robustness but also decreases accuracy noticeably. It is not clear whether there is a certain fundamental limit on achieving robustness whilst maintaining accuracy. By adopting Bayes error to robustness analysis, we investigate the limit of certified robust accuracy.
  arXiv  Detail & Related papers  (2024-05-19T13:23:05Z)
• Selecting Learnable Training Samples is All DETRs Need in Crowded Pedestrian Detection [72.97320260601347]
  In crowded pedestrian detection, the performance of DETRs is still unsatisfactory due to the inappropriate sample selection method. We propose Sample Selection for Crowded Pedestrians, which consists of the constraint-guided label assignment scheme (CGLA) Experimental results show that the proposed SSCP effectively improves the baselines without introducing any overhead in inference.
  arXiv  Detail & Related papers  (2023-05-18T08:28:01Z)
• Confidence-aware Training of Smoothed Classifiers for Certified Robustness [75.95332266383417]
  We use "accuracy under Gaussian noise" as an easy-to-compute proxy of adversarial robustness for an input. Our experiments show that the proposed method consistently exhibits improved certified robustness upon state-of-the-art training methods.
  arXiv  Detail & Related papers  (2022-12-18T03:57:12Z)
• Uncertainty-Driven Action Quality Assessment [11.958132175629368]
  We propose a novel probabilistic model, named Uncertainty-Driven AQA (UD-AQA), to capture the diversity among multiple judge scores.<n>We generate the estimation of uncertainty for each prediction, which is employed to re-weight AQA regression loss.<n>Our proposed method achieves competitive results on three benchmarks including the Olympic events MTL-AQA and FineDiving, and the surgical skill JIGSAWS datasets.
  arXiv  Detail & Related papers  (2022-07-29T07:21:15Z)
• Labeling-Free Comparison Testing of Deep Learning Models [28.47632100019289]
  We propose a labeling-free comparison testing approach to overcome the limitations of labeling effort and sampling randomness. Our approach outperforms the baseline methods by up to 0.74 and 0.53 on Spearman's correlation and Kendall's $tau$, regardless of the dataset and distribution shift.
  arXiv  Detail & Related papers  (2022-04-08T10:55:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.