Does the Same Token Mean the Same State? MoE Routing as Signal for Reasoning Control

• URL: http://arxiv.org/abs/2606.22798v1
• Date: Mon, 22 Jun 2026 03:17:31 GMT
• Title: Does the Same Token Mean the Same State? MoE Routing as Signal for Reasoning Control
• Abstract summary: We operationalize textbfRAD (Routing Agreement Decoding) as an answer-string-free multi-rollout selector.<n>It locates a fixed anchor, represents each rollout by its anchor-window MoE routing states, and returns the densest Weighted-Jaccard $K$-NN route-basin center.
• Score: 53.43997385459622
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In sparse Mixture-of-Experts language models, does the same token id imply the same router state and the same experts producing it? Holding the emitted token id fixed at repeated anchors, we find it does not: the experts that produce it still separate task context, trajectory history, and reasoning-effort mode. This residual structure supports test-time control: near \emph{boundary} anchors (the final-response transition) and \emph{delimiter} anchors (which open the answer, e.g.\ \texttt{\textbackslash boxed\{} or code fences), routing neighborhoods already align with final-answer basins at a marker-only readout and strongest when the routing is read at the answer opening. We operationalize this as \textbf{RAD} (Routing Agreement Decoding), an answer-string-free multi-rollout selector: it locates a fixed anchor, represents each rollout by its anchor-window MoE routing states, and returns the densest Weighted-Jaccard $K$-NN route-basin center, without parsing, normalizing, executing, or voting over answer strings. Across 10 sparse-MoE configurations (gpt-oss, Qwen3-MoE) and 6 datasets spanning math, GPQA, and code, RAD is on par with Majority where string voting is well-posed, with small positive paired deltas (RAD $73.9$ / RAD+DC $74.2$ vs.\ Majority $73.6$). Like majority voting, RAD is not a verifier: a dense \emph{wrong} basin can still win. Its value is the interface: the same selector gives direct pass@1 on code, where exact-string voting is ill-defined, and the same routing-density principle, re-anchored to the agentic boundary, improves best-of-16 patch selection on SWE-bench Verified over random, where patches have no answer string to vote on.

Related papers

• Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents [60.650808962786364]
  We study whether native MoE router traces can guide steering and selection without an external judge or selection-time test execution.<n>Our analysis shows that routing provides a robust behavioral role signal; token-granular readouts and decision-matched comparison sets turn it into effective control.<n>We evaluate on SWE-bench Verified using open-weight sparse MoE agents across scales and reasoning-effort settings.
  arXiv  Detail & Related papers  (2026-08-23T03:07:03Z)
• Directional Contextual Representations for Dependency Relations: Why Cross-Direction Pairing Fails [22.49571926071395]
  We show that pairing a token's forward state against a emphcandidate's backward state (cross-direction'' pairing, $F_i$ vs. $B_j$) consistently emphunderperforms same-direction pairing, and the penalty emphgrows, not shrinks, with token distance, both paired-bootstrap significant.
  arXiv  Detail & Related papers  (2026-08-21T00:50:25Z)
• Long-Horizon State Tracking in LLMs: Executing MD5 through a Deep Sequence of Dependent Tool Calls [0.5097809301149342]
  Long-horizon tasks remain uncommon in large language model (LLM) evaluation.<n>We test whether an LLM can carry exact intermediate state across many tool calls at all.<n>Two ingredients decide success: keeping the model's own reasoning in its context each turn, and voting over a thinking-enabled worker to remove its modular-arithmetic slips.
  arXiv  Detail & Related papers  (2026-08-02T09:38:02Z)
• Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routing [65.59464371673623]
  Local Branch Routing (LBR) is a token-level test-time scaling framework.<n>It expands a small local lookahead tree, forwards all sampled branches through the language model, and uses a lightweight router to select the depth-1 subtree to commit.<n>LBR allows each token decision to use evidence beyond the root next-token distribution while avoiding full solution-level search.
  arXiv  Detail & Related papers  (2026-06-24T03:42:44Z)
• What Context Does a Coding Agent Actually Need to Act? [0.0]
  A modern coding agent can hold an entire repository in its context window.<n>We study when the agent must emphedit code.<n>We score context against real issue resolution on SWE-bench Verified.
  arXiv  Detail & Related papers  (2026-06-19T04:22:17Z)
• Isolating LLM Alignment from Regex: Zero Coverage and Metric-Dependent Divergence Under Adversarial Mutation [51.56484100374058]
  Previous work found no measurable coverage gain from adding a live Gemini backend behind an active filter.<n>We introduce $L_5$-no-regex -- identical to $L_4$-real (Gemini-2.5-flash, token-budget cap, rate limit, output scrub) but with the nine-pattern filter disabled.<n>We evaluate it against adversarial probes across three sub-languages.
  arXiv  Detail & Related papers  (2026-06-12T18:54:24Z)
• When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models [27.942734943134983]
  Mixture-of-Experts (MoE) language models route each token to a small subset of experts.<n>We compare each standard route against sampled equal-compute alternatives for the same token and score each by the next-token probability it assigns to the realized token in a verified reasoning trajectory.<n>The result is sharply token-conditional: the standard router is well-aligned with route utility on confident tokens but uninformative on the fragile tokens that drive hard reasoning.
  arXiv  Detail & Related papers  (2026-05-08T05:26:09Z)
• AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation [56.21301367698041]
  We introduce AnchorSeg, which reformulates reasoning segmentation as a structured conditional generation process over image tokens.<n>We model spatial conditioning as a factorized distribution over image tokens, where the anchor query determines localization signals.<n>We propose Token--Mask Cycle Consistency (TMCC), a bidirectional training objective that enforces alignment across resolutions.
  arXiv  Detail & Related papers  (2026-04-20T17:49:22Z)
• PAR$^2$-RAG: Planned Active Retrieval and Reasoning for Multi-Hop Question Answering [57.89576196160413]
  Large language models (LLMs) remain brittle on multi-hop question answering (MHQA)<n>We propose textbfPlanned Active Retrieval and Reasoning RAG (PAR$2$-RAG), a two-stage framework that separates emphcoverage from emphcommitment.
  arXiv  Detail & Related papers  (2026-03-30T23:52:54Z)
• Think in Parallel, Answer as One: Logit Averaging for Open-Ended Reasoning [102.13989674248116]
  ThinkMerge is a training-free, plug-and-play decoding strategy.<n>It runs K parallel reasoning traces and averages their next-token logits at synchronization points to produce a single coherent output.
  arXiv  Detail & Related papers  (2025-12-02T15:35:31Z)
• Natural Language Edge Labelling: Decoupling Intent from Execution in Structured LM Reasoning [0.0]
  We introduce Natural Language Edge Labelling (NLEL), a labeller-tuner overlay that attaches a free-form natural-language directive to each search edge.<n>We show NLEL strictly generalizes CoT/ToT, prove an anytime-monotonicity property for top-$k$ selection under labelconditioned bundles, and bound selector shortfall by controlvector distortion.
  arXiv  Detail & Related papers  (2025-10-06T14:00:02Z)
• Auditable Early Stopping for Agentic Routing: Ledger-Verified Run-Wise Certificates under Local DP [0.0]
  We address when a best-first router for tool-use agents can stop exploring without missing a better leaf.<n>We introduce a run-wise certificate that couples each node's key to the same exponential race that realizes leaf perturbations.<n>Experiments on synthetic graphs and a small real pipeline show tight stopping, deterministic replay, and low overhead.
  arXiv  Detail & Related papers  (2025-09-09T01:25:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.