Task Structure Reverses Layerwise State Encoding in Sequence Models

• URL: http://arxiv.org/abs/2606.00926v1
• Date: Sat, 30 May 2026 23:26:29 GMT
• Title: Task Structure Reverses Layerwise State Encoding in Sequence Models
• Abstract summary: We find that the same architecture reverses this profile when the task changes.<n>In fine-tuned Mamba-130M and Pythia-160M, Parity is concentrated late in Mamba and the recurrent baselines and built gradually by Transformer.<n>The same flip appears in fine-tuned Mamba-130M and Pythia-160M, and the Pythia Dyck bottleneck persists at 410M.
• Score: 8.403971471573607
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mechanistic studies of sequence models often treat layerwise state encodings as architectural traits: recurrent models concentrate readable state, attention-based models distribute it. We find that the same architecture reverses this profile when the task changes. Across Transformers, Mamba, Mamba-2, LSTMs, and GRUs, Parity is concentrated late in Mamba and the recurrent baselines and built gradually by Transformer; on bounded-depth Dyck-k the pattern flips. The same flip appears in fine-tuned Mamba-130M and Pythia-160M, and the Pythia Dyck bottleneck persists at 410M. Two explanations are conflated in the literature: algebraic structure (commutativity) versus computational structure (prefix update vs. stack). To separate them we add a third task: non-commutative S_3 permutation composition. S_3 groups with Parity, not Dyck, on layerwise probing across all five architectures and on Mamba-specific Conv1D attribution, so the grouping tracks computational structure rather than commutativity. Causal interventions show that, in the 4-layer formal models, linearly readable directions are often functionally necessary and can remain important at out-of-distribution lengths on Parity and Dyck. At pretrained scale the picture splits. Fine-tuned Pythia Dyck has a strong middle-layer bottleneck (L6-L7 ablation drops accuracy by roughly 81% at 160M; broader L4-L18 plateau at 410M), far weaker at the best-probe layer. Pretrained Mamba shows the complementary failure mode: its final layer is highly readable, no single probe direction breaks the task on Parity, Dyck, or S_3, yet mid-position activation patching there recovers about 97-98% of the clean-corrupted logit gap. Probing localizes where state is linearly available, not always where the computation is bottlenecked. Mechanistic signatures are properties of architecture and task together.

Related papers

• Pre-carved Niches: The Formation Dynamics of Modular Task Partitions in Early LLM Training [2.446942686520097]
  Large language models exhibit a modular internal organization that mirrors well-studied functional networks of the human brain.<n>We train a Pythia-410M model from scratch (two trajectories, bf16 and fp32) and run attribution patching at every step.<n> deviation from the substrate appears only in the domain being learned, consistent with the hypothesis that modularity tracks learning.
  arXiv  Detail & Related papers  (2026-09-01T12:47:09Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• BluTrain: A C++/CUDA Framework for AI Systems [2.79727705929128]
  BluTrain is a robust, lightweight, architecture-general training framework in standard C++ and the core programming model.<n>In formal evaluations BluTrain outperforms industry-standard baselines in both throughput and memory efficiency.<n>With every layer explicitly open to native tuning, the performance ceiling is the framework's own to raise.
  arXiv  Detail & Related papers  (2026-06-23T16:39:25Z)
• Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory [1.081571058570587]
  AlltoAll dispatch is the dominant bottleneck of MoE expert parallelism.<n>We introduce DODOCO to test two assumptions about the workload.<n> scaling EP changes the per-expert max/mean token ratio by at most 5% within every architecture's measurable range.
  arXiv  Detail & Related papers  (2026-05-20T10:14:00Z)
• Density Field State Space Models: 1-Bit Distillation, Efficient Inference, and Knowledge Organization in Mamba-2 [0.0]
  We present Density Field State Space Models (DF-SSM), a framework for compressing SSMs to a 1-bit scaffold with int8 low-rank correction.<n>We achieve a 278 MB model (9.7x smaller than the 2.7 GB FP16 teacher) that runs at 21.4x faster inference on GPU.
  arXiv  Detail & Related papers  (2026-04-28T14:25:10Z)
• Three Roles, One Model: Role Orchestration at Inference Time to Close the Performance Gap Between Small and Large Agents [0.4666493857924357]
  We study whether inference-time scaffolding alone, without any additional training compute, can improve the performance of a small model in complex multi-step environments.<n>We evaluate Qwen3-8B on the AppWorld benchmark under both full-precision and 4-bit quantized configurations.<n>On full-precision inference, our scaffolded 8B model surpasses DeepSeek-Coder 33B Instruct (7.1%) from the original AppWorld evaluation.
  arXiv  Detail & Related papers  (2026-04-13T13:40:33Z)
• Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study [0.0]
  Cross-architecture evaluation of production LLM inference on AMD Instinct MI325X GPUs.<n> benchmarking four models spanning 235B to 1 trillion parameters across three architectural families.
  arXiv  Detail & Related papers  (2026-02-27T13:21:48Z)
• How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding [3.8914132324834045]
  Chain-of-thought (CoT) prompting boosts Large Language Models accuracy on multi-step tasks.<n>But whether the generated "thoughts" reflect the true internal reasoning process is unresolved.<n>We present the first feature-level causal study of CoT faithfulness.
  arXiv  Detail & Related papers  (2025-07-24T10:25:46Z)
• Exploring Diffusion Transformer Designs via Grafting [82.91123758506876]
  We present grafting, a simple approach for editing pretrained diffusion transformers (DiTs) to materialize new architectures under small compute budgets.<n>We show that new diffusion model designs can be explored by grafting pretrained DiTs, with edits ranging from operator replacement to architecture restructuring.
  arXiv  Detail & Related papers  (2025-06-05T17:59:40Z)
• Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners [72.37408197157453]
  Recent advancements have demonstrated that the performance of large language models (LLMs) can be significantly enhanced by scaling computational resources at test time.<n>This raises a fundamental question: can models with lower complexity leverage their superior generation throughput to outperform similarly sized Transformers for a fixed computational budget?<n>To address this question and overcome the lack of strong subquadratic reasoners, we distill pure and hybrid Mamba models from pretrained Transformers.
  arXiv  Detail & Related papers  (2025-02-27T18:08:16Z)
• Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity [56.0251572416922]
  State Space Models (SSMs) have emerged as efficient alternatives to Transformers for sequential modeling.<n>We propose a novel SSM architecture that introduces modality-aware sparsity through modality-specific parameterization of the Mamba block.<n>We evaluate Mixture-of-Mamba across three multi-modal pretraining settings.
  arXiv  Detail & Related papers  (2025-01-27T18:35:05Z)
• The Mamba in the Llama: Distilling and Accelerating Hybrid Models [76.64055251296548]
  We show how to distill large Transformers into linear RNNs by reusing the linear projection weights from attention layers with academic GPU resources.<n>The resulting hybrid model achieves performance comparable to the original Transformer in chat benchmarks.<n>We also introduce a hardware-aware speculative decoding algorithm that accelerates the inference speed of Mamba and hybrid models.
  arXiv  Detail & Related papers  (2024-08-27T17:56:11Z)
• Compute Better Spent: Replacing Dense Layers with Structured Matrices [77.61728033234233]
  We identify more efficient alternatives to dense matrices, as exemplified by the success of convolutional networks in the image domain. We show that different structures often require drastically different initialization scales and learning rates, which are crucial to performance. We propose a novel matrix family containing Monarch matrices, the Block-Train, which we show performs better than dense for the same compute on multiple tasks.
  arXiv  Detail & Related papers  (2024-06-10T13:25:43Z)
• Efficient Content-Based Sparse Attention with Routing Transformers [34.83683983648021]
  Self-attention suffers from quadratic compute and memory requirements with respect to sequence length. Our work proposes to learn dynamic sparse attention patterns that avoid allocating and memory to attend to content unrelated to the query of interest.
  arXiv  Detail & Related papers  (2020-03-12T19:50:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.