Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers

• URL: http://arxiv.org/abs/2603.11114v1
• Date: Wed, 11 Mar 2026 12:45:53 GMT
• Title: Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers
• Abstract summary: We introduce routing signatures, a vector representation summarizing expert activation patterns across layers for a given prompt.<n>We show that prompts from the same task category induce highly similar routing signatures, while prompts from different categories exhibit substantially lower similarity.<n>A logistic regression trained solely on routing signatures achieves 92.5% +/- 6.1% cross-validated accuracy on four-way task classification.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse Mixture-of-Experts (MoE) architectures enable efficient scaling of large language models through conditional computation, yet the routing mechanisms responsible for expert selection remain poorly understood. In this work, we introduce routing signatures, a vector representation summarizing expert activation patterns across layers for a given prompt, and use them to study whether MoE routing exhibits task-conditioned structure. Using OLMoE-1B-7B-0125-Instruct as an empirical testbed, we show that prompts from the same task category induce highly similar routing signatures, while prompts from different categories exhibit substantially lower similarity. Within-category routing similarity (0.8435 +/- 0.0879) significantly exceeds across-category similarity (0.6225 +/- 0.1687), corresponding to Cohen's d = 1.44. A logistic regression classifier trained solely on routing signatures achieves 92.5% +/- 6.1% cross-validated accuracy on four-way task classification. To ensure statistical validity, we introduce permutation and load-balancing baselines and show that the observed separation is not explained by sparsity or balancing constraints alone. We further analyze layer-wise signal strength and low-dimensional projections of routing signatures, finding that task structure becomes increasingly apparent in deeper layers. These results suggest that routing in sparse transformers is not merely a balancing mechanism, but a measurable task-sensitive component of conditional computation. We release MOE-XRAY, a lightweight toolkit for routing telemetry and analysis.

Related papers

• Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents [60.650808962786364]
  We study whether native MoE router traces can guide steering and selection without an external judge or selection-time test execution.<n>Our analysis shows that routing provides a robust behavioral role signal; token-granular readouts and decision-matched comparison sets turn it into effective control.<n>We evaluate on SWE-bench Verified using open-weight sparse MoE agents across scales and reasoning-effort settings.
  arXiv  Detail & Related papers  (2026-08-23T03:07:03Z)
• Agent-as-a-Router: Agentic Model Routing for Coding Tasks [54.00153674547733]
  Agent-as-a-router (AC) formalizes routing as a C-A-F loop (Context->Action->FeedbackContext)<n>AC achieves the lowest cumulative regret on in-distribution tasks and generalizes to out-of-distribution agentic-programming tasks.
  arXiv  Detail & Related papers  (2026-06-22T06:37:31Z)
• Signals: Trajectory Sampling and Triage for Agentic Interactions [1.6631602844999722]
  We propose a lightweight, signal-based framework for triaging agentic interaction trajectories.<n>Our approach computes cheap, broadly applicable signals from live interactions and attaches them as structured attributes for trajectory triage.<n>We show that signal-based sampling achieves an 82% informativeness rate compared to 74% for filtering and 54% for random sampling.
  arXiv  Detail & Related papers  (2026-04-01T01:08:07Z)
• When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models [0.0]
  We identify a routing paradox in hybrid recurrent-attention architectures.<n>We show that content-based routing requires exactly the pairwise computation that routing is designed to avoid.
  arXiv  Detail & Related papers  (2026-03-22T01:04:57Z)
• Path-Constrained Mixture-of-Experts [23.80197713200086]
  MoE routing selects each layer's experts independently, creating NL possible expert paths -- for N experts across L layers.<n>We propose pathmoe, which shares router parameters across consecutive layers.<n>Experiments on 0.9B and 16B parameter models demonstrate consistent improvements on perplexity and downstream tasks over independent routing.
  arXiv  Detail & Related papers  (2026-03-18T21:35:53Z)
• TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training [53.93696896939915]
  Training tool-use agents typically rely on Supervised Fine-Tuning (SFT) on successful trajectories and Reinforcement Learning (RL) on pass-rate-selected tasks.<n>We propose TopoCurate, an interaction-aware framework that projects multi-trial rollouts from the same task into a unified semantic quotient topology.<n>TopoCurate achieves consistent gains of 4.2% (SFT) and 6.9% (RL) over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-03-02T10:38:54Z)
• Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts [32.65737144630759]
  Mixture-of-Experts (MoE) architectures scale large language models efficiently by employing a parametric "router" to dispatch tokens to a sparse subset of experts.<n>We introduce kNN-MoE, a retrieval-augmented routing framework that reuses optimal expert assignments from a memory of similar past cases.<n>Experiments show kNN-MoE outperforms zero-shot baselines and rivals computationally expensive supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-01-05T14:16:11Z)
• Route-DETR: Pairwise Query Routing in Transformers for Object Detection [11.46025964297103]
  Detection Transformer (DETR) offers an end-to-end solution for object detection.<n>DETR suffers from inefficient query competition where multiple queries converge to similar positions.<n>We present Route-DETR, which addresses these issues through adaptive pairwise routing in decoder self-attention layers.
  arXiv  Detail & Related papers  (2025-12-15T20:26:58Z)
• Automated Circuit Interpretation via Probe Prompting [0.0]
  We present probe prompting, an automated pipeline that transforms attribution graphs into compact, interpretable subgraphs.<n>Across five prompts, probe-prompted subgraphs preserve high explanatory coverage while compressing complexity.<n>Compared to geometric clustering baselines, concept-aligned groups exhibit higher behavioral coherence.
  arXiv  Detail & Related papers  (2025-11-10T11:53:36Z)
• RouteMark: A Fingerprint for Intellectual Property Attribution in Routing-based Model Merging [69.2230254959204]
  We propose RouteMark, a framework for IP protection in merged MoE models.<n>Our key insight is that task-specific experts exhibit stable and distinctive routing behaviors under probing inputs.<n>For attribution and tampering detection, we introduce a similarity-based matching algorithm.
  arXiv  Detail & Related papers  (2025-08-03T14:51:58Z)
• vCache: Verified Semantic Prompt Caching [95.16654660556975]
  This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees.<n>It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training.<n>Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines.
  arXiv  Detail & Related papers  (2025-02-06T04:16:20Z)
• How not to Stitch Representations to Measure Similarity: Task Loss Matching versus Direct Matching [0.7366405857677227]
  We argue that task loss matching may be very misleading as a similarity index.<n>It can indicate very high similarity between very distant layers, whose representations are known to have different functional properties.<n>Even more surprisingly, when comparing layers within the same network, task loss matching often indicates that some layers are more similar to a layer than itself.
  arXiv  Detail & Related papers  (2024-12-15T20:18:49Z)
• Semantic-Aligned Matching for Enhanced DETR Convergence and Multi-Scale Feature Fusion [95.7732308775325]
  The proposed DEtection TRansformer (DETR) has established a fully end-to-end paradigm for object detection. DETR suffers from slow training convergence, which hinders its applicability to various detection tasks. We design Semantic-Aligned-Matching DETR++ to accelerate DETR's convergence and improve detection performance.
  arXiv  Detail & Related papers  (2022-07-28T15:34:29Z)
• StableMoE: Stable Routing Strategy for Mixture of Experts [109.0602120199226]
  Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead. We propose StableMoE with two training stages to address the routing fluctuation problem. Results show that StableMoE outperforms existing MoE methods in terms of both convergence speed and performance.
  arXiv  Detail & Related papers  (2022-04-18T16:48:19Z)
• Learning to Generate Content-Aware Dynamic Detectors [62.74209921174237]
  We introduce a newpective of designing efficient detectors, which is automatically generating sample-adaptive model architecture. We introduce a course-to-fine strat-egy tailored for object detection to guide the learning of dynamic routing. Experiments on MS-COCO dataset demonstrate that CADDet achieves 1.8 higher mAP with 10% fewer FLOPs compared with vanilla routing.
  arXiv  Detail & Related papers  (2020-12-08T08:05:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.