p-Spin Glass Network Efficient Single-Batch Continual Learning

• URL: http://arxiv.org/abs/2608.14774v1
• Date: Fri, 14 Aug 2026 16:24:37 GMT
• Title: p-Spin Glass Network Efficient Single-Batch Continual Learning
• Abstract summary: We introduce the $p$-Spin Glass Network, a novel architecture that overcomes limitations, structurally manages optimization variance and yields four noticeable capabilities.<n>It demonstrates sample efficiency, matching the performance of a Transformer baseline while utilizing $8times$ fewer training sequences.<n>Ultimately, this work removes large requirement for stable deep learning, establishing a foundation for continuous learning and edge AI.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern sequence models heavily rely on massive memory footprints and large-batch stochastic optimization, barriers that restrict sample efficiency and continual learning. We introduce the $p$-Spin Glass Network, a novel architecture that overcomes these limitations, structurally manages optimization variance and yields four noticeable capabilities: 1. It enforces memory efficiency: native ternary quantization compresses internal parameters by $8\times$, while exact implicit gradients strictly bound activation memory to $\mathcal{O}(B \cdot T \cdot D)$. 2. it demonstrates sample efficiency, matching the asymptotic performance of a Transformer baseline while utilizing $8\times$ fewer training sequences. 3. Method enables single-batch stability and smooth, monotonic convergence at a stochastic micro-batch size of $1$. 4. Finally, this stability proves modality-agnostic, maintaining robust temporal credit assignment across both discrete subword and long horizon uncompressed raw byte streams. Ultimately, this work removes large batch requirement for stable deep learning, establishing a foundation for continuous learning and edge AI.

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond [50.440302567029654]
  Multi-modal intelligence has made the Key-Value cache a dominant memory bottleneck for efficient deployment.<n>In this work, we revisit the inherent limitations of the per-channel quantization paradigm.<n>We propose OScaR, an accurate and lightweight KV cache compression framework for X-LLMs.
  arXiv  Detail & Related papers  (2026-05-19T10:53:03Z)
• DexWorldModel: Causal Latent World Modeling towards Automated Learning of Embodied Tasks [54.32016216994156]
  We introduce the Causal Latent World Model (CLWM), which employs DINOv3 features as generative targets to disentangle interaction semantics from visual noise.<n>To overcome memory scaling, CLWM features a Dual-State Test-Time Training (TTT) Memory that guarantees a strict $mathcalO(1)$ footprint for long-horizon tasks.<n>We present EmbodiChain, an online framework that establishes the Efficiency Law by injecting an infinite flow of physics-grounded trajectories during training.
  arXiv  Detail & Related papers  (2026-04-13T03:19:36Z)
• 6Bit-Diffusion: Inference-Time Mixed-Precision Quantization for Video Diffusion Models [32.45415555978057]
  Diffusion transformers have demonstrated remarkable capabilities in generating videos.<n>However, their practical deployment is constrained by high memory usage and computational cost.<n>Post-Training Quantization provides a practical way to reduce memory usage and boost speed.
  arXiv  Detail & Related papers  (2026-03-19T10:45:11Z)
• Omni-Masked Gradient Descent: Memory-Efficient Optimization via Mask Traversal with Improved Convergence [5.683382109687174]
  Memory-efficient optimization methods have recently gained increasing attention for scaling full-tuning of of large language models under the GPU-memory bottleneck.<n>We propose Omni-O-Masked Descent (OMGD) as an approach to memory-efficient training.
  arXiv  Detail & Related papers  (2026-03-06T06:47:52Z)
• Beyond Scattered Acceptance: Fast and Coherent Inference for DLMs via Longest Stable Prefixes [10.877713536966601]
  Longestahead Prefix (LSP) scheduler is a training-free and model-agnostic inference paradigm based on monolithic prefix absorption.<n>LSP evaluates token stability via a single forward pass, dynamically identifies a contiguous left-aligned block of stable predictions.<n>It snaps its boundary to natural linguistic or structural acceptances before an atomic commitment.
  arXiv  Detail & Related papers  (2026-03-05T18:25:26Z)
• RaBiT: Residual-Aware Binarization Training for Accurate and Efficient LLMs [5.782015253162346]
  Residual binarization enables matmul-free inference by stacking binary layers.<n>We propose RaBiT, a novel quantization framework that resolves coadaptation by algorithmically enforcing a residual hierarchy.<n>RaBiT achieves state-of-the-art performance, rivals even hardware-intensive Vector Quantization (VQ) methods, and delivers a $4.49times$ inference speed-up over full-precision models.
  arXiv  Detail & Related papers  (2026-02-05T06:41:11Z)
• Beyond Softmax: A Natural Parameterization for Categorical Random Variables [61.709831225296305]
  We introduce the $textitcatnat$ function, a function composed of a sequence of hierarchical binary splits.<n>A rich set of experiments show that the proposed function improves the learning efficiency and yields models characterized by consistently higher test performance.
  arXiv  Detail & Related papers  (2025-09-29T12:55:50Z)
• VAMO: Efficient Zeroth-Order Variance Reduction for SGD with Faster Convergence [6.574641780732972]
  Large-scale non problems are common in deep learning.<n>First-order (FO)s serve as today's baselines.<n>ZO algorithms reduce the size of computation and memory costs.<n>VAMO achieves these gains with smaller dynamic memory requirements.
  arXiv  Detail & Related papers  (2025-05-20T05:31:15Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)
• On Convergence of Incremental Gradient for Non-Convex Smooth Functions [63.51187646914962]
  In machine learning and network optimization, algorithms like shuffle SGD are popular due to minimizing the number of misses and good cache. This paper delves into the convergence properties SGD algorithms with arbitrary data ordering.
  arXiv  Detail & Related papers  (2023-05-30T17:47:27Z)
• 8-bit Optimizers via Block-wise Quantization [57.25800395197516]
  Statefuls maintain statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past values. This state can be used to accelerate optimization compared to plain gradient descent but uses memory that might otherwise be allocated to model parameters. In this paper, we develop first gradients that use 8-bit statistics while maintaining the performance levels of using 32-bit gradient states.
  arXiv  Detail & Related papers  (2021-10-06T15:43:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.