Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction

• URL: http://arxiv.org/abs/2604.00733v2
• Date: Sun, 05 Apr 2026 12:23:35 GMT
• Title: Spectral Compact Training: Pre-Training Large Language Models via Permanent Truncated SVD and Stiefel QR Retraction
• Abstract summary: We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) VT.<n>SCT achieves up to 199x memory reduction per layer at rank 32, enabling full training steps on a Steam Deck handheld.<n>Rank-sweep experiments on SmolLM2-1.7B show that all tested ranks converge to the same loss floor (4.2-4.5), identifying the learning rate schedule -- not rank -- as the primary bottleneck.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The memory wall remains the primary bottleneck for training large language models on consumer hardware. We introduce Spectral Compact Training (SCT), a method that replaces dense weight matrices with permanent truncated SVD factors W = U diag(s) V^T, where the full dense matrix is never materialized during training or inference. Gradients flow through the compact spectral factors via standard backpropagation, and U, V are retracted to the Stiefel manifold via QR decomposition after each optimizer step. SCT achieves up to 199x memory reduction per MLP layer at rank 32, enabling full training steps of 70B-parameter architectures on a Steam Deck handheld (7.2 GB peak memory vs. 1,245 GB for dense FP32 training with Adam). Rank-sweep experiments on SmolLM2-1.7B (ranks 32-256, 2000 steps, NVIDIA A100) show that all tested ranks converge to the same loss floor (~4.2-4.5), identifying the learning rate schedule -- not MLP rank -- as the primary bottleneck. Rank 128 emerges as the efficiency sweet spot at 11.7x MLP compression with the lowest perplexity. GPU memory drops 46% at rank 32 while training throughput doubles.

Related papers

• Per-Matrix Optimality Is Not Enough: Three-Level Optimization for Low-Rank LLM Compression [8.890261387138215]
  We introduce a three-level chain that widens optimization scope from individual matrices to Transformer blocks to the full model.<n>On LLaMA-7B at 60% compression, the chain reduces WikiText-2 perplexity from 42.1 to 19.1 to 11.4.<n> skipping the block-level stage becomes competitive, revealing an offline compute--data trade-off.
  arXiv  Detail & Related papers  (2026-09-14T16:36:46Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Eddy-VL 1.9B: Structural Pruning and Layered Distillation for Edge-Deployable Multimodal Embedding [1.2132884030603328]
  We introduce Eddy-VL 1.9B, a compressed multimodal embedding model built on Qwen3-VL-Embedding-2B for offline, edge-deployable vision-language retrieval.<n>Eddy-VL targets air-gapped forensic and investigative settings where cloud APIs are unavailable and low latency is essential.<n>The released model contains 1,926,188,032 parameters (3.85 GB bf16), representing approximately 9.5% fewer parameters than the 2.13B teacher model.
  arXiv  Detail & Related papers  (2026-07-15T04:36:19Z)
• FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training [43.28268986528884]
  FORGE folds step into the backward pass and applies it one tile at a time, entirely in registers, so each tile is consumed the instant it is produced and never becomes a tensor.<n>For every element-wise update, for every element-wise rule, for every element-wise rule, for every exactness survives and sequence-paralleling.
  arXiv  Detail & Related papers  (2026-06-22T07:08:31Z)
• AdpSplit: Error-Driven Adaptive Splitting for Faster Geometry Discovery in 3D Gaussian Splatting [61.977804325410695]
  We introduce AdpSplit, an error-driven adaptive split operator that determines the number of split children and initializes the child parameters from L1-pixel-error region statistics.<n>Across the MipNeRF360, Deep-Blending, and Tanks&Temples datasets, AdpSplit reduces the training time of multiple accelerated 3DGS pipelines by 9.2%-22.3%.<n>With FastGS, AdpSplit matches the full-schedule PSNR on MipNeRF360 while reducing training time by 16.4%, corresponding to a 12.6x acceleration over vanilla 3DGS.
  arXiv  Detail & Related papers  (2026-05-07T19:23:16Z)
• Lightweight Distillation of SAM 3 and DINOv3 for Edge-Deployable Individual-Level Livestock Monitoring and Longitudinal Visual Analytics [0.3609538870261841]
  Foundation-model pipelines for individual-level livestock monitoring have raised the accuracy ceiling of precision livestock farming.<n>But their GPU memory budgets exceed the envelope of commodity edge accelerators.<n>To close this gap, the backbone of SAM 3 is distilled into a 40.66M-scale multi-scale student.<n>The pipeline achieves 97.34% top-1 accuracy with 91.67% macro-F1 on nine-class pig behaviour classification.
  arXiv  Detail & Related papers  (2026-04-29T19:25:22Z)
• MeanFlow Transformers with Representation Autoencoders [71.45823902973349]
  MeanFlow (MF) is a diffusion-motivated generative model that enables efficient few-step generation by learning long jumps directly from noise to data.<n>We develop an efficient training and sampling scheme for MF in the latent space of a Representation Autoencoder (RAE)<n>We achieve a 1-step FID of 2.03, outperforming vanilla MF's 3.43, while reducing sampling GFLOPS by 38% and total training cost by 83% on ImageNet 256.
  arXiv  Detail & Related papers  (2025-11-17T06:17:08Z)
• Speeding Up MACE: Low-Precision Tricks for Equivarient Force Fields [51.95157731126864]
  Machine-learning force fields can deliver accurate molecular dynamics (MD) at high computational cost.<n>This thesis aims to make MACE cheaper and faster by identifying computational bottlenecks and evaluating low-precision execution policies.
  arXiv  Detail & Related papers  (2025-10-23T14:02:34Z)
• ARA: Adaptive Rank Allocation for Efficient Large Language Model SVD Compression [23.58843227762227]
  In large language model (LLM) compression, singular value decomposition (SVD) is a widely studied and adopted low-rank decomposition technique.<n>Under a global compression ratio constraint, determining the appropriate rank for different linear modules becomes a critical problem.<n>We propose an Adaptive Rank Allocation (ARA) method to address this problem.
  arXiv  Detail & Related papers  (2025-10-22T09:05:47Z)
• PreLoRA: Hybrid Pre-training of Vision Transformers with Full Training and Low-Rank Adapters [2.5547655072779]
  We propose an approach to identify states of partial convergence and switch from full parameter training to Low-Rank Adaptation (LoRA) on the ViT-Large model.<n> Experimental results show that this approach preserves model accuracy while reducing the number of trainable parameters to 10% of its original size.
  arXiv  Detail & Related papers  (2025-09-25T21:34:17Z)
• Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss [59.835032408496545]
  We propose a tile-based strategy that partitions the contrastive loss calculation into arbitrary small blocks. We also introduce a multi-level tiling strategy to leverage the hierarchical structure of distributed systems. Compared to SOTA memory-efficient solutions, it achieves a two-order-of-magnitude reduction in memory while maintaining comparable speed.
  arXiv  Detail & Related papers  (2024-10-22T17:59:30Z)
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model [118.06260386652778]
  We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs.
  arXiv  Detail & Related papers  (2024-05-07T15:56:43Z)
• GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection [133.45193150403537]
  Training Large Language Models (LLMs) presents significant memory challenges due to the growing size of weights and GPU states. In this work, we propose Gradient Low-Rank Projection (GaLore) as a memory-efficient training strategy. Our 8-bit GaLore further reduces memory by up to 82.5% and total training memory by 63.3%, compared to a BF16 baseline.
  arXiv  Detail & Related papers  (2024-03-06T07:29:57Z)
• Reduced Precision Floating-Point Optimization for Deep Neural Network On-Device Learning on MicroControllers [15.37318446043671]
  This paper introduces a novel reduced precision optimization technique for On-Device Learning (ODL) primitives on MCU-class devices. Our approach results more than two orders of magnitude faster than existing ODL software frameworks for single-core MCUs.
  arXiv  Detail & Related papers  (2023-05-30T16:14:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.