One-for-All: A Lightweight Stabilized and Parameter-Efficient Pre-trained LLM for Time Series Forecasting

• URL: http://arxiv.org/abs/2603.29756v1
• Date: Tue, 31 Mar 2026 13:54:43 GMT
• Title: One-for-All: A Lightweight Stabilized and Parameter-Efficient Pre-trained LLM for Time Series Forecasting
• Abstract summary: One-for-All is a framework for adapting pre-trained Large Language Models for time-series analysis.<n>rsLoRA introduces a mathematically grounded rank-stabilization mechanism that enables provable gradient stability at low ranks.<n>One-for-All achieves state-of-the-art efficiency-accuracy trade-offs.
• Score: 4.364999214109123
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We address the challenge of adapting pre-trained Large Language Models (LLMs) for multivariate time-series analysis, where their deployment is often hindered by prohibitive computational and memory demands. Our solution, One-for-All, introduces Gaussian Rank-Stabilized Low-Rank Adapters (rsLoRA) to enable parameter-efficient fine-tuning of frozen LLMs. While inspired by LoRA, rsLoRA introduces a mathematically grounded rank-stabilization mechanism that enables provable gradient stability at low ranks a novel contribution absent in prior PEFT methods. Our framework injects trainable rank decomposition matrices (rank 16) into positional embeddings and output layers, while keeping self-attention weights fixed. This design reduces trainable parameters by 6.8$\times$ (vs. TimesNet), 21$\times$ (vs. GPT4TS), and 11.8$\times$ (vs. TIME-LLM), while achieving a 168-1,776$\times$ smaller memory footprint (2.2MiB vs. 340MiB-4.18GiB in SOTA models). Rigorous evaluation across six time-series tasks demonstrates that One-for-All achieves state-of-the-art efficiency-accuracy trade-offs: 5.5$\times$ higher parameter efficiency (MSE=5.50) than TimesNet and 21$\times$ better than GPT4TS, while matching their forecasting accuracy (MSE=0.33). The framework's stability is validated through consistent performance across diverse horizons (96-720 steps) and datasets (ETT, Weather, M3, M4), with 98.3% fewer parameters than conventional transformers. These advances enable deployment on edge devices for healthcare, finance, and environmental monitoring without compromising performance.

Related papers

• Frozen Cores Need Task Signal: Fisher-Whitened Cross-Covariance for Low-Resource LLM Adaptation [18.8104643146881]
  We study the choice of constructor-efficient fine-tuning under a severe trainable-state budget.<n>We introduce FCCA, which estimates the signed input--error cross-covariance, whitens it with diagonal Fisher moments, truncates it in the resulting local metric, maps the selected directions back, and applies thin QR to obtain stable core coordinates.<n>On Qwen2.5-3B, FCCA reaches an 83.0 macro-average, 2.3 points above the next-best matched-budget constructor, and exceeds its unwhitened RawGrad control on all 11 tasks.
  arXiv  Detail & Related papers  (2026-09-01T05:52:38Z)
• How Small Can You Go? A Controlled Study of LoRA Rank, Target Modules, and Quantization Trade-offs for Text-to-SQL on a 60M-Parameter Model [0.0]
  We show that LoRA with r=16 recovers within 11.6 percentage points of full fine-tuning accuracy (59.6% vs. 71.2% exact-match) while training fewer than 1% of parameters and consuming 31% less peak GPU memory.<n>QLoRA with INT8 and NF4 quantization achieves comparable accuracy (52.8% and 53.2%) at dramatically lower memory cost (0.60 GB each)
  arXiv  Detail & Related papers  (2026-07-28T11:12:03Z)
• SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning [0.0]
  We introduce Hankel Reduced order Model (HRM) adapter, an SSM-based residual module for such tasks.<n>By leveraging the time-invariance of the system matrix $barA$, HRM enables an exact FFT-based parallel scan, achieving computational parity with LoRA.<n> Gate analysis reveals that HRM effectively learns to modulate recurrence, providing a robust architectural alternative to low-rank adaptation for long-context sequence modeling.
  arXiv  Detail & Related papers  (2026-06-24T18:36:31Z)
• Kernelized Sparse Fine-Tuning with Bi-level Parameter Competition for Vision Models [80.50996301430108]
  sparse tuning achieves remarkable performance by adjusting only the weights most relevant to downstream tasks.<n>We propose a one-stage method named SNELLA to overcome the above limitations.<n> SNELLA achieves SOTA performance with low memory usage.
  arXiv  Detail & Related papers  (2025-10-28T03:39:18Z)
• EfficientLLM: Efficiency in Large Language Models [64.3537131208038]
  Large Language Models (LLMs) have driven significant progress, yet their growing counts and context windows incur prohibitive compute, energy, and monetary costs.<n>We introduce EfficientLLM, a novel benchmark and the first comprehensive empirical study evaluating efficiency techniques for LLMs at scale.
  arXiv  Detail & Related papers  (2025-05-20T02:27:08Z)
• SPAT: Sensitivity-based Multihead-attention Pruning on Time Series Forecasting Models [8.817690876855728]
  We propose a structured pruning method, SPAT ($textbfS$ensitivity $textbfP$runer for $textbfAt$tention), which selectively removes attention mechanisms and yields highly effective models.<n>Experiments on datasets demonstrate that SPAT-pruned models achieve reductions of 2.842% in MSE, 1.996% in MAE, and 35.274% in FLOPs.
  arXiv  Detail & Related papers  (2025-05-13T17:39:31Z)
• Dynamic Low-Rank Sparse Adaptation for Large Language Models [54.1231638555233]
  Low-rank Sparse Adaptation (LoSA) is a novel method that seamlessly integrates low-rank adaptation into sparse LLM sparsity.<n>LoSA dynamically sparsifies the LoRA outcomes based on the corresponding sparse weights during fine-tuning.<n>LoSA can efficiently boost the efficacy of sparse LLMs within a few hours, without introducing any additional inferential burden.
  arXiv  Detail & Related papers  (2025-02-20T18:37:32Z)
• Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation [17.649599011681712]
  Large Language Models (LLMs) are highly resource-intensive to fine-tune due to their enormous size.<n>This paper highlights the importance of effective parameterization in low-rank fine-tuning to reduce estimator variance and enhance the stability of final model outputs.<n>We propose MonteCLoRA, an efficient fine-tuning technique that employs Monte Carlo estimation to learn an unbiased posterior estimation of low-rank parameters with low expected variance.
  arXiv  Detail & Related papers  (2024-11-07T01:31:48Z)
• Structured Unrestricted-Rank Matrices for Parameter Efficient Fine-tuning [38.80020737321214]
  We propose a framework for efficient parameter fine-tuning (PEFT) based on structured unrestricted-rank matrices (SURM)<n>SURMs achieve 5-7% accuracy gains on various image classification tasks while replacing low-rank matrices in LoRA.<n>It also results in up to 12x reduction of the number of parameters in adapters (with virtually no loss in quality) on the GLUE benchmark.
  arXiv  Detail & Related papers  (2024-06-25T17:26:05Z)
• MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning [71.50432879573614]
  Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional.<n>We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank.<n>Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks.
  arXiv  Detail & Related papers  (2024-02-27T07:14:12Z)
• HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference [68.59839755875252]
  HiRE comprises of two novel components: (i) a compression scheme to cheaply predict top-$k$ rows/columns with high recall, followed by full computation restricted to the predicted subset, and (ii) DA-TOP-$k$: an efficient multi-device approximate top-$k$ operator. We demonstrate that on a one billion parameter model, HiRE applied to both the softmax as well as feedforward layers, achieves almost matching pretraining and downstream accuracy, and speeds up inference latency by $1.47times$ on a single TPUv5e device.
  arXiv  Detail & Related papers  (2024-02-14T18:04:36Z)
• APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference [63.52244442498831]
  Fine-tuning and inference with large Language Models (LMs) are generally known to be expensive. We introduce APT that adaptively prunes and tunes parameters for the LMs. We show that APT speeds up LMs fine-tuning by up to 8x and reduces large LMs memory training footprint by up to 70%.
  arXiv  Detail & Related papers  (2024-01-22T18:39:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.