Layer-wise MoE Routing Locality under Shared-Prefix Code Generation: Token-Identity Decomposition and Compile-Equivalent Fork Redundancy

• URL: http://arxiv.org/abs/2604.17182v1
• Date: Sun, 19 Apr 2026 00:56:08 GMT
• Title: Layer-wise MoE Routing Locality under Shared-Prefix Code Generation: Token-Identity Decomposition and Compile-Equivalent Fork Redundancy
• Abstract summary: In LLM-based code generation, multiple code candidates are often generated in parallel from the same prompt.<n>We study Qwen3.5-35B-A3B-FP8 by performing tree-search-based branching generation from a shared prefix.<n>We show that diversity in top-P search, including beam search, poses a significant challenge.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In LLM-based code generation, multiple code candidates are often generated in parallel from the same prompt -- for example, in best-of-N sampling or multi-candidate code completion. These requests can share KV caches through a common prefix, yet the extent to which their Mixture-of-Experts (MoE) expert routing overlaps, and how this overlap varies across layers, remains insufficiently understood. We study Qwen3.5-35B-A3B-FP8 (256 routed experts, top-8) by performing tree-search-based branching generation from a shared prefix (851 completed codes, temperature 0.7) and analyzing the results with a compiler-output-based alignment (gcc -S -O0 assembly) that controls for token-identity confounds. Our findings are threefold: (1) At positions where both sequences generated the same token, Jaccard similarity reaches 0.649 (40x random), while even at positions with different tokens it remains 0.175 (11x random). (2) A layer-wise decomposition reveals a crossing pattern: same-token routing similarity exceeds different-token similarity across all layers, but dips in the middle layers (L14-20), while different-token similarity peaks in the middle layers at 14x random. (3) In tree-search code generation, 67% of successfully compiled codes concentrate in the top three assembly-equivalent groups, and 99.6% of within-group differences consist of comments and blank lines. We show that diversity in top-P search, including beam search, poses a significant challenge. These results refine the "context-independent routing" claim of prior work through layer-wise decomposition and suggest opportunities for improving search efficiency in LLM code generation.

Related papers

• Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems [67.04448659688579]
  We develop a query-independent 16-entry codebook that ranks the top decoded candidates in a single batched forward pass.<n>With no iterative search and no message passing at test time, Codebook Agent is the most accurate method on all six benchmarks.
  arXiv  Detail & Related papers  (2026-09-02T08:10:22Z)
• Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routing [65.59464371673623]
  Local Branch Routing (LBR) is a token-level test-time scaling framework.<n>It expands a small local lookahead tree, forwards all sampled branches through the language model, and uses a lightweight router to select the depth-1 subtree to commit.<n>LBR allows each token decision to use evidence beyond the root next-token distribution while avoiding full solution-level search.
  arXiv  Detail & Related papers  (2026-06-24T03:42:44Z)
• Memory-Guided Tree Search with Cross-Branch Knowledge Transfer for LLM Solver Synthesis [3.0663322946413287]
  We introduce MEMOIR, a memory-guided tree-search framework with a two-level memory hierarchy.<n>Across seven Combinatorial optimization problems, MEMOIR achieves 96.7% solution validity.<n>MeMOIR's run-to-run validity standard deviation is more than an order of magnitude below that of every baseline we evaluated.
  arXiv  Detail & Related papers  (2026-05-17T16:47:31Z)
• Mining Subscenario Refactoring Opportunities in Behaviour-Driven Software Test Suites: ML Classifiers and LLM-Judge Baselines [1.9537983097153042]
  Behaviour-Driven Development (BDD) software test suites accumulate duplicated step subsequences.<n>Three published patterns are available (within-file Background, within-repo reusable-scenario invocation, cross-organisational shared higher-level step)<n>No prior work automates which recurring subsequences are worth extracting or which mechanism applies.
  arXiv  Detail & Related papers  (2026-05-14T08:38:04Z)
• MARS$^2$: Scaling Multi-Agent Tree Search via Reinforcement Learning for Code Generation [29.92851142870157]
  Reinforcement learning (RL) paradigms have demonstrated strong performance on reasoning-intensive tasks such as code generation.<n>We propose textbfMARS$2$ (Multi-Agent Reinforced Tree-Search Scaling), a unified RL framework in which multiple independently-optimized agents collaborate.
  arXiv  Detail & Related papers  (2026-04-16T02:52:24Z)
• Divide and Conquer: Accelerating Diffusion-Based Large Language Models via Adaptive Parallel Decoding [6.755667885643806]
  Diffusion-based large language models (dLLMs) have shown promising performance across various reasoning tasks.<n>We introduce an adaptive parallel decoding approach, namely DiCo, which features a three-phase divide-and-conquer paradigm.<n>Extensive experiments demonstrate that DiCo can achieve significant inference speedups while maintaining competitive generation quality.
  arXiv  Detail & Related papers  (2026-02-27T08:36:06Z)
• Hierarchical Embedding Fusion for Retrieval-Augmented Code Generation [6.4453302264198165]
  We present Hierarchical Embedding Fusion (HEF), a two-stage approach to repository representation for code completion.<n>HEF achieves exact-match accuracy comparable to snippet-based retrieval baselines.<n>Compared to graph-based and iterative retrieval systems, HEF reduces median end-to-end latency by 13 to 26 times.
  arXiv  Detail & Related papers  (2026-02-04T14:56:11Z)
• Improving Diffusion Language Model Decoding through Joint Search in Generation Order and Token Space [110.80564213032729]
  Diffusion Language Models (DLMs) offer order-agnostic generation that can explore many possible decoding trajectories.<n>We introduce Order-Token Search to explore this space through jointly searching over generation order and token values.
  arXiv  Detail & Related papers  (2026-01-28T07:55:07Z)
• Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable Rewards [48.321707628011005]
  Lookahead Tree-Based Rollouts (LATR) is a novel rollout strategy designed to explicitly promote trajectory-level diversity.<n>LATR accelerates policy learning by 131% on average and improves final pass@1 performance by 4.2%.
  arXiv  Detail & Related papers  (2025-10-28T11:12:02Z)
• CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models [106.11371409170818]
  Large language models (LLMs) can act as agents with capabilities to self-refine and improve generated code autonomously. We propose CodeTree, a framework for LLM agents to efficiently explore the search space in different stages of the code generation process. Specifically, we adopted a unified tree structure to explicitly explore different coding strategies, generate corresponding coding solutions, and subsequently refine the solutions.
  arXiv  Detail & Related papers  (2024-11-07T00:09:54Z)
• Hierarchical Matching and Reasoning for Multi-Query Image Retrieval [113.44470784756308]
  We propose a novel Hierarchical Matching and Reasoning Network (HMRN) for Multi-Query Image Retrieval (MQIR) It disentangles MQIR into three hierarchical semantic representations, which is responsible to capture fine-grained local details, contextual global scopes, and high-level inherent correlations. Our HMRN substantially surpasses the current state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-06-26T07:03:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.