From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference

• URL: http://arxiv.org/abs/2607.28097v1
• Date: Thu, 30 Jul 2026 12:07:56 GMT
• Title: From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference
• Abstract summary: Expert-reduction orders can produce observably different sparse-MoE executions.<n>We isolate this effect in native DeepSeek-V4-Flash by freezing local MoE state and varying only aggregation semantics.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mathematically equivalent expert-reduction orders can produce observably different sparse-MoE executions. We isolate this effect in native DeepSeek-V4-Flash by freezing local MoE state and varying only aggregation semantics. Four schemes separate operand representation from accumulator precision. At one layer-5 fork, 720 A-mode orders yield 10 continuation basins; 720 B-mode orders form 360 exact structural classes and 11 basins. Under one Chinese prompt, the B classes split into 202 layoffs, 113 hiring, and 45 other continuations. Maximum-L-infinity B-branch selection separates 12, 24, and 36 of 50 prompts by 8, 16, and 32 tokens. Across 192 persistent trajectories per scheme, P32, A, and B change every native-reference route trajectory, while C preserves routes, token sequences, and texts. A separate 192-trajectory C check matches native MoE, post-mHC, next-router, and LM states bitwise. For one controlled B branch, exact post-mHC endpoint reconstruction reproduces the measured downstream trajectory. At the next decode boundary, exact FP64 reconstruction of the branch's full persistent state yields agreement for 301 downstream post-mHC states, 301 persistent-state checkpoints, 301 routes, predictions, and text over seven steps, given the same naturally generated next input. These controls identify post-mHC as an intra-token boundary and full persistent state as a cross-token continuation boundary. Identical tokens need not imply identical autoregressive state: divergence can survive a token boundary and become visible later. These results make expert operand conversion, accumulator precision, and reduction order part of a numerical compatibility contract for sparse-MoE runtimes and hardware backends. They establish controlled causal possibility, not deployment incidence; C's order invariance is limited to evaluated six-term states and schedules.

Related papers

• SeGDeP: Semantic- and Geometric-Aware Decoupled Prompts for Reasoning Segmentation [51.46828526392219]
  Reasoning segmentation converts an implicit linguistic conclusion into a precise mask.<n>Existing MLLM-segmenter interfaces either use a special trigger or compress both signals into one context.<n>We present SeGDeP, an explicit what-where interface.
  arXiv  Detail & Related papers  (2026-09-08T15:11:32Z)
• SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models [53.86918766240095]
  Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation.<n>We propose textbfState Anomaly Neutralization (SANE), which applies adaptive $tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure.
  arXiv  Detail & Related papers  (2026-08-23T10:41:07Z)
• The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference [0.10152838128195468]
  Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable.<n>We test that assumption: holding the checkpoint, prompts, hardware, inference engine, decoding, and quantization configuration fixed, we swap only the INT8 linear kernel inside vLLM.<n>At 1.7B each arm reproduces itself bit-for-bit across cold restarts, yet the arms agree on no sequence in any end-to-end comparison we ran (0/8, 0/16, and 0/64).
  arXiv  Detail & Related papers  (2026-08-13T20:34:36Z)
• QuoteBench: How Matched Scores Can Hide Command-Path Failures [30.480838412827907]
  Execution scores alone cannot distinguish command-generation errors from failures introduced after generation.<n> QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families.<n> Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state rather than treat a matched score as an intrinsic model property.
  arXiv  Detail & Related papers  (2026-08-13T17:57:20Z)
• Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation [51.56484100374058]
  Stage-replay reconstructs intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix.<n>A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides.
  arXiv  Detail & Related papers  (2026-07-30T16:41:40Z)
• PULSE: An Executable Contract Language for Spatiotemporal Knowledge Graph Engineering [2.2262915480980063]
  We present PULSE, an Object-Process-Methodology-inspired language that localizes four operational roles and their write effects in one typed runtime.<n>The implemented contract fixes evidence non-overwrite, branch isolation, grounded multisubject timers, guarded state change, and declaration-ranked event ordering over time and space.<n>GeoSPARQL, SOSA, and SHACL remain generated views; Lean 4 checks kernel analogues for positions, evidence, clocks, monitors, atomicity, and branch source retention.
  arXiv  Detail & Related papers  (2026-07-26T17:44:05Z)
• Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation [0.0]
  We document a failure class in frontier large language models.<n>The failure reproduces at first observation, but its empirical surface is unstable.<n>For regulated, frontier-model accuracy is a moving compliance boundary that shifts without notice.
  arXiv  Detail & Related papers  (2026-07-25T22:40:02Z)
• Operational Proto-Introspection in Looped Language Models: Process-Quality Taps, Executable Branching, and the Readout-Control Boundary [0.0]
  Can a language model read the quality of its ongoing computation?<n>Can an external intervention turn that readout into better outcomes?<n>We test both questions in a frozen 2.6B looped transformer, Ouro-RLTT.
  arXiv  Detail & Related papers  (2026-07-20T22:40:36Z)
• Constrained Decoding for Diffusion Language Models via Efficient Inference over Finite Automata [57.27430779838529]
  We present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton.<n>The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding.<n> Empirical evaluations on Dream-7B and LLaDA-8 show substantial accuracy gains across various tasks.
  arXiv  Detail & Related papers  (2026-07-08T05:48:57Z)
• Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity [45.829192030010006]
  We show that most mass concentrates in local block-diagonal neighborhoods with a light cross-block residue.<n>We derive a blockwise evaluation of a resolvent-style operator that keeps within-block interactions exact and routes cross-block interactions through a reduced system.<n>Our method matches the dense operator's accuracy while reducing wall-clock time by $12-29%$ under a standardized measurement protocol.
  arXiv  Detail & Related papers  (2026-05-21T13:35:48Z)
• Reading Calibrated Uncertainty from Language Model Trajectories [46.663987199083245]
  Methods that probe the model's internal activations feed raw hidden states into opaque snapshots and leave implicit the layer-wise trajectory by which a representation is formed.<n>We extract eleven scale-invariant geometric features, tracing the cumulative path of per-layer updates, and feed them to a sparse linear probe.<n>The probe outperforms MSP under selective abstention, with baseline scaling with up to 21 AURC points.
  arXiv  Detail & Related papers  (2026-05-19T19:24:29Z)
• Measuring Maximum Activations in Open Large Language Models [60.3514350516308]
  We measure global and layerwise maxima on 27 checkpoints from 8 open families spanning dense, MoE, vision-language, intermediate-training, and instruction-tuned variants.<n>We conclude that maximum activation magnitude is a model property tied to family, architecture, and training stage - not a simple byproduct of size.
  arXiv  Detail & Related papers  (2026-05-15T03:31:51Z)
• CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling [46.16066322190728]
  We introduce the Continuous Acoustic Wave Network (CAWN), a fully continuous sequence-mixing architecture.<n>Instead of discrete matrix-based attention, CAWN projects hidden states into multi-headed complex-domain phasors.<n>To prevent signal degradation over ultra-long contexts, we introduce a dual-gated Selective Phase Resonance mechanism.
  arXiv  Detail & Related papers  (2026-04-05T20:13:22Z)
• BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models [56.504879072674015]
  We propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients.<n>BPDQ enables serving Qwen2.5-72B on a single GTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit)
  arXiv  Detail & Related papers  (2026-02-04T02:54:37Z)
• Arcee: Differentiable Recurrent State Chain for Generative Vision Modeling with Mamba SSMs [3.5590835022089813]
  State-space models (SSMs) are increasingly adopted for long-context sequence modeling.<n>Recent "Mamba-for-vision" variants largely explore multiple scan orders to relax strict causality for non-sequential signals.<n>Arcee is a cross-block recurrent state chain that reuses each block's terminal state-space representation.
  arXiv  Detail & Related papers  (2025-11-14T12:44:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.