Parcae: Scaling Laws For Stable Looped Language Models

• URL: http://arxiv.org/abs/2604.12946v1
• Date: Tue, 14 Apr 2026 16:43:37 GMT
• Title: Parcae: Scaling Laws For Stable Looped Language Models
• Abstract summary: Traditional fixed-depth architectures scale quality by increasing training FLOPs, typically through increased parameterization, at the expense of a higher memory footprint, or data.<n>A potential alternative is looped architectures, which instead increase FLOPs by sending activations through a block of layers in a loop.<n>While promising, existing recipes for training looped architectures can be unstable, suffering from residual explosion and loss spikes.<n>We propose Parcae, a novel stable, looped architecture that constrains the spectral norm of the injection parameters via discretization of a negative diagonal parameterization.
• Score: 35.9547796403241
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional fixed-depth architectures scale quality by increasing training FLOPs, typically through increased parameterization, at the expense of a higher memory footprint, or data. A potential alternative is looped architectures, which instead increase FLOPs by sending activations through a block of layers in a loop. While promising, existing recipes for training looped architectures can be unstable, suffering from residual explosion and loss spikes. We address these challenges by recasting looping as a nonlinear time-variant dynamical system over the residual stream. Via a linear approximation to this system, we find that instability occurs in existing looped architectures as a result of large spectral norms in their injection parameters. To address these instability issues, we propose Parcae, a novel stable, looped architecture that constrains the spectral norm of the injection parameters via discretization of a negative diagonal parameterization. As a result, Parcae achieves up to 6.3% lower validation perplexity over prior large-scale looped models. Using our stable looped architecture, we investigate the scaling properties of looping as a medium to improve quality by increasing FLOPs in training and test-time. For training, we derive predictable power laws to scale FLOPs while keeping parameter count fixed. Our initial scaling laws suggest that looping and data should be increased in tandem, given a fixed FLOP budget. At test-time, we find that Parcae can use looping to scale compute, following a predictable, saturating exponential decay. When scaled up to 1.3B parameters, we find that Parcae improves CORE and Core-Extended quality by 2.99 and 1.18 points when compared to strong Transformer baselines under a fixed parameter and data budget, achieving a relative quality of up to 87.5% a Transformer twice the size.

Related papers

• SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers [50.00050394179417]
  Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size.<n>We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache.<n>Through a series of ablations, we arrive at a recipe we call SMELT, which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets.
  arXiv  Detail & Related papers  (2026-09-01T14:52:57Z)
• RecurrentGPT: Expressive Depth through Recurrent Modulation in Transformers [8.085475675888045]
  RecurrentGPT is a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times.<n>Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate to modulate the recurrent update.<n>Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality.
  arXiv  Detail & Related papers  (2026-08-15T06:22:00Z)
• Looped State-Space Language Models with Adaptive Exit-State Selection [43.98434319030707]
  Recent work on looped language models suggests that many reasoning problems benefit from greater computational depth rather than additional independent parameters.<n>We investigate Looped Mamba and Looped Hybrid Mamba-Transformer architectures, which repeatedly apply a shared Mamba block to introduce explicit finite-depth recurrent computation.<n>We adapt Ouro's two-stage exit gate to Looped Mamba for threshold-controlled selection among recurrent-step outputs.
  arXiv  Detail & Related papers  (2026-07-11T04:09:19Z)
• The Key to Going Linear: Analysis-Driven Transformer Linearization [8.56204304015324]
  We show that softmax relies on key-dependent, rank-1 projections, elucidating why delta-style networks outperform purely gated accumulation.<n>We identify a potential source of approximation errors and introduce structural interventions, specifically sink tokens, short convolutions, and fixed-budget cache routing.
  arXiv  Detail & Related papers  (2026-07-08T17:59:09Z)
• Simply Stabilizing the Loop via Fully Looped Transformer [41.240805541680395]
  Looped Transformer suffers from training instability when the number of loop iterations increases.<n>Our experiments demonstrate that Fully Looped Transformer improves training stability, enhances downstream performance, and provides preliminary adaptability under different test-time compute budgets.
  arXiv  Detail & Related papers  (2026-05-11T07:21:53Z)
• Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments [31.754045125599305]
  Plateaus in certain regimes arise because sample-based estimates of the loss become poor proxies for the true objective over the course of training.<n>We show that there are two ways to address this type of learning stagnation: either reduce the step size or increase the number of samples collected between updates.<n>We vastly outperform prior baselines in a complex openended domain by scaling PPO to more than 1M parallel environments.
  arXiv  Detail & Related papers  (2026-03-06T08:07:08Z)
• LoopViT: Scaling Visual ARC with Looped Transformers [14.9105267508928]
  We propose Loop-ViT, which decouples reasoning depth from model capacity through weight-tied recurrence.<n>Loop-ViT iterates a weight-tied Hybrid Block, combining local convolutions and global attention, to form a latent chain of thought.<n> Empirical results on the ARC-AGI-1 benchmark validate this perspective.
  arXiv  Detail & Related papers  (2026-02-02T14:32:57Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• Large Continual Instruction Assistant [59.585544987096974]
  Continual Instruction Tuning (CIT) is adopted to instruct Large Models to follow human intent data by data.<n>Existing update gradient would heavily destroy the performance on previous datasets during CIT process.<n>We propose a general continual instruction tuning framework to address the challenge.
  arXiv  Detail & Related papers  (2024-10-08T11:24:59Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• Deep Equilibrium Optical Flow Estimation [80.80992684796566]
  Recent state-of-the-art (SOTA) optical flow models use finite-step recurrent update operations to emulate traditional algorithms. These RNNs impose large computation and memory overheads, and are not directly trained to model such stable estimation. We propose deep equilibrium (DEQ) flow estimators, an approach that directly solves for the flow as the infinite-level fixed point of an implicit layer.
  arXiv  Detail & Related papers  (2022-04-18T17:53:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.