S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models

• URL: http://arxiv.org/abs/2604.01168v2
• Date: Fri, 03 Apr 2026 19:40:10 GMT
• Title: S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models
• Abstract summary: tuning a single initial state matrix per recurrent layer with zero overhead outperforms LoRA by +10.8 pp (p wenSpider on HumanEval)<n> tuning method we call S0 tuning improves greedy pass@1 by +23.6 +/- 1.7 pp (10 seeds)<n>Cross-domain transfer is significant on MATH-500 (+4.8 pp, p = 0.00002, 8 seeds) and GSM8K = 0.0003, 10 seeds.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Using roughly 48 execution-verified HumanEval training solutions, tuning a single initial state matrix per recurrent layer, with zero inference overhead, outperforms LoRA by +10.8 pp (p < 0.001) on HumanEval. The method, which we call S0 tuning, optimizes one state matrix per recurrent layer while freezing all model weights. On Qwen3.5-4B (GatedDeltaNet hybrid), S0 tuning improves greedy pass@1 by +23.6 +/- 1.7 pp (10 seeds). On FalconH1-7B (Mamba-2 hybrid), S0 reaches 71.8% +/- 1.3 and LoRA reaches 71.4% +/- 2.4 (3 seeds), statistically indistinguishable at this sample size while requiring no weight merging. Cross-domain transfer is significant on MATH-500 (+4.8 pp, p = 0.00002, 8 seeds) and GSM8K (+2.8 pp, p = 0.0003, 10 seeds); a text-to-SQL benchmark (Spider) shows no transfer, consistent with the trajectory-steering mechanism. A prefix-tuning control on a pure Transformer (Qwen2.5-3B) degrades performance by -13.9 pp under all nine configurations tested. On Qwen3.5, a per-step state-offset variant reaches +27.1 pp, above both S0 and LoRA but with per-step inference cost. Taken together, the results show that recurrent state initialization is a strong zero-inference-overhead PEFT surface for hybrid language models when verified supervision is scarce. The tuned state is a ~48 MB file; task switching requires no weight merging or model reload. Code and library: https://github.com/jackyoung27/s0-tuning.

Related papers

• Schema-Adaptive Action-Conditioned JEPA for Cross-Machine CNC Transfer under Partial Sensor Overlap [0.0]
  Cross-machine deployment of industrial world models requires transfer across changes in dynamics, sensing interfaces, sampling regimes, and control units.<n>We study a schema-adaptive action-conditioned Joint-Embedding Predictive Architecture (SAAC-JEPA) for CNC dynamics.<n> Evaluation uses group-disjoint source splits, source-only normalization, held-out self-supervised validation, unit audits, and a sealed target test after model locking.
  arXiv  Detail & Related papers  (2026-09-13T15:10:06Z)
• ActionSplice: In-Flight Action Editing for Interactive World Models [4.400011068855375]
  Chunk-autoregressive video world models typically condition each generated chunk on one action.<n>We introduce Actionplice, an inference framework that formulates this problem as Counterfactual State Transport (CST)<n>A temporal-splicing variant $mathrmCST*R$ preserves a temporal prefix and updates only the suffix.<n>$mathrmCST*T$ reduces suffix LPIPS by 56.1% and 77.5%, respectively, while providing $2.73times$ and $1.69times$ pixel-ready speedups over waiting.
  arXiv  Detail & Related papers  (2026-09-08T04:21:36Z)
• Adapting Without Gradients: Affine Statistics Transport and What Its Certificate Can Tell You [0.8594140167290097]
  Test-time adaptation assumes that model parameters can be updated at inference time.<n>We study adaptation when the learned model must remain frozen.<n>We introduce CASTER, a method that stores source class statistics in a discriminative subspace.
  arXiv  Detail & Related papers  (2026-08-31T21:07:36Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• DREAM Technical Report [53.37525197047151]
  We present DREAM, an autonomous optimization control architecture that adds a perception-aware, orchestrable, and auditable policy layer atop existing pipelines without replacing them.<n> DREAM has two core components. First, a three-tier Intent Engine fuses on-device signals into structured L0/L1/L2 intent representations; its edge-cloud trigger chain reduces reporting volume to approximately 8.7%.<n>Second, a Meta Engine uses a MetaModel for layered M1-to-M2-to-M3 reasoning: intent summarization, strategy planning informed by Strategy Memory, and parameter translation. It dispatches the resulting parameters through a unified outlet with
  arXiv  Detail & Related papers  (2026-08-10T10:35:32Z)
• LLM-Driven Neural Network Generation with Same-Family Architecture Guidance: Disentangling Transfer and Adaptation [48.83701310501069]
  Large language models (LLMs) can generate neural-network modifications, but unrestricted generation is often invalid or harmful.<n>This paper studies a narrower setting: improving a weak target model using a stronger same-family source model from a neural-network database.<n>We propose a source-guided candidate-generation protocol with non-source controls, source-conditioned candidates, and a no-LLM hp_copy ablation under equal evaluation budgets.
  arXiv  Detail & Related papers  (2026-07-06T23:59:18Z)
• CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMs [0.0]
  We introduce CRMA, a residual adapter whose internal mixing matrix M is doubly-stochastic at every forward pass via Sinkhorn normalization.<n>On Mistral-7B across 5 domains 3 seeds, modular per-task LoRA mediates a backbone loss-relative drift from +42.96% +/- 5.5 (naive fine-tuning) to -0.17% +/- 0.17, with disjoint per-seed ranges, and improves prior-task holdout loss by 1.99% +/- 0.54 over a matched frozen-substrate baseline.
  arXiv  Detail & Related papers  (2026-05-29T21:50:23Z)
• Benchmarking Patent Embeddings: A Multi-Task Evaluation of 22 Models Across Retrieval, Classification, and Clustering [0.0]
  We evaluate 22 pre-trained embedding models on three tasks -- information retrieval, classification, and clustering.<n>We find that two results cast doubt on the prevailing wisdom.
  arXiv  Detail & Related papers  (2026-05-22T23:51:13Z)
• Measuring Maximum Activations in Open Large Language Models [60.3514350516308]
  We measure global and layerwise maxima on 27 checkpoints from 8 open families spanning dense, MoE, vision-language, intermediate-training, and instruction-tuned variants.<n>We conclude that maximum activation magnitude is a model property tied to family, architecture, and training stage - not a simple byproduct of size.
  arXiv  Detail & Related papers  (2026-05-15T03:31:51Z)
• Scaling DoRA: High-Rank Adaptation via Factored Norms and Fused Kernels [83.99688944263843]
  Weight-De Low-Rank Adaptation (DoRA) extends LoRA by decoupling weight magnitude from direction.<n>At d_in = 8192 and rank r = 384, a single module's norm requires about 512 MB of transient working memory in bf16.<n>A factored norm decomposes the squared norm into base, cross, and Gram terms computable through O(d_out r + r2) intermediates, eliminating the dense product.
  arXiv  Detail & Related papers  (2026-03-23T17:57:24Z)
• Token Coherence: Adapting MESI Cache Protocols to Minimize Synchronization Overhead in Multi-Agent LLM Systems [0.0]
  Multi-agent LLM orchestration incurs synchronization costs scaling as O(n x S x |D|) in agents, steps, and artifact size under naive broadcast.<n>I argue this pathology is a structural residue of full-state rebroadcast, not an inherent property of multi-agent coordination.<n>I construct the Artifact Coherence System (ACS) and prove the Token Coherence Theorem.
  arXiv  Detail & Related papers  (2026-03-16T12:20:06Z)
• ACAR: Adaptive Complexity Routing for Multi-Model Ensembles with Auditable Decision Traces [3.151184728006369]
  We present ACAR, a measurement framework for studying multi-model orchestration under auditable conditions.<n>ACAR uses self-consistency variance (sigma) computed from N=3 probe samples to route tasks across single-model, two-model, and three-model execution modes.<n>We evaluate ACAR on 1,510 tasks spanning four benchmarks, producing more than 7,550 auditable runs.
  arXiv  Detail & Related papers  (2026-02-06T23:27:17Z)
• LiQSS: Post-Transformer Linear Quantum-Inspired State-Space Tensor Networks for Real-Time 6G [85.58816960936069]
  Proactive and agentic control in Sixth-Generation (6G) Open Radio Access Networks (O-RAN) requires control-grade prediction under stringent Near-Time (Near-RT) latency and computational constraints.<n>This paper investigates a post-Transformer paradigm for efficient radio telemetry forecasting.<n>We propose a quantum-inspired state-space tensor network that replaces self-attention with stable structured state-space dynamics kernels.
  arXiv  Detail & Related papers  (2026-01-18T12:08:38Z)
• Kernelized Sparse Fine-Tuning with Bi-level Parameter Competition for Vision Models [80.50996301430108]
  sparse tuning achieves remarkable performance by adjusting only the weights most relevant to downstream tasks.<n>We propose a one-stage method named SNELLA to overcome the above limitations.<n> SNELLA achieves SOTA performance with low memory usage.
  arXiv  Detail & Related papers  (2025-10-28T03:39:18Z)
• POME: Post Optimization Model Edit via Muon-style Projection [74.73326657229347]
  Post-Optimization Model Edit (POME) enhances the performance of fine-tuned large language models.<n>It uses a muon-style projection to $Delta W$, the difference between the fine-tuned and pretrained weights.<n>As a simple post-processing step, POME is completely decoupled from the training pipeline.
  arXiv  Detail & Related papers  (2025-10-08T04:20:11Z)
• Synthetic Data RL: Task Definition Is All You Need [27.637296092731322]
  Reinforcement learning (RL) is a powerful way to adapt foundation models to specialized tasks, but its reliance on large-scale human-labeled data limits broad adoption.<n>We introduce Synthetic Data RL, a simple and general framework that reinforcement fine-tunes models using only synthetic data generated from a task definition.<n>Our method first generates question and answer pairs from the task definition and retrieved documents, then adapts the difficulty of the question based on model solvability, and selects questions using the average pass rate of the model across samples for RL training.
  arXiv  Detail & Related papers  (2025-05-18T05:35:13Z)
• S*: Test Time Scaling for Code Generation [55.11863577956177]
  We propose S*, the first hybrid test-time scaling framework for code generation.<n>S* substantially improves the coverage and selection accuracy of generated code.
  arXiv  Detail & Related papers  (2025-02-20T09:18:53Z)
• Layer-Adaptive State Pruning for Deep State Space Models [1.5749416770494706]
  We provide a structured pruning method for SSMs, Layer-Adaptive STate pruning (LAST)<n>Last scores are evaluated using the $mathcalH_infty$ norms of subsystems and layer-wise energy normalization.<n>We demonstrate that, on average, pruning 33% of states still maintains performance with 0.52% accuracy loss in multi-input multi-output SSMs without retraining.
  arXiv  Detail & Related papers  (2024-11-05T05:50:51Z)
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch [72.97553348776425]
  We introduce DARE to set most delta parameters without affecting the abilities of Supervised Fine-Tuning (SFT) LMs. Then, we use DARE as a versatile plug-in to sparsify delta parameters of multiple SFT models and merge them into a single model. We also utilize DARE to create a merged LM that ranks first among models with 7 billion parameters on the Open Leaderboard.
  arXiv  Detail & Related papers  (2023-11-06T13:43:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.