The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability

• URL: http://arxiv.org/abs/2607.20301v1
• Date: Wed, 22 Jul 2026 15:48:39 GMT
• Title: The Blessing of Dimensionality: How Near-Orthogonality in High-Dimensional Spaces Explains Temporal Portability
• Abstract summary: We study the long-term temporal portability of PortLLM patches across 10 continual pretraining steps using base models Mistral, Gemma, and Qwen.<n>We find that the portability persists across longer time duration, indicating that repeated fine-tuning is not required when the base model is periodically updated.
• Score: 39.02435736839143
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Fine-tuning has been widely used to adapt large language models (LLMs) for domain-specific tasks. Parameter efficient fine-tuning (PEFT) methods such as low-rank adaptation (LoRA) are frequently used to reduce computational costs. PortLLM is a training-free and data-free scheme used to adapt LLMs after continual pretraining. Although the initial PortLLM results show that LoRA patches exhibit short-term temporal portability, the long-term performance of PortLLM across several updates of continual pretraining remains underexplored. Furthermore, the intriguing effectiveness of PortLLM is not well understood from a theoretical standpoint. We address these two open questions by (1) performing an extensive empirical study of the long-term temporal portability of PortLLM patches across 10 continual pretraining steps using base models Mistral, Gemma, and Qwen; and (2) offering two theoretical analyses to explain our observation that the simple PortLLM method achieves competitive performance. We find empirically that the portability persists across longer time duration, indicating that repeated fine-tuning is not required when the base model is periodically updated. We find theoretically that near-orthogonality of high-dimensional vectors is a key justification for temporal portability. Our analyses also demonstrate a geometric perspective of the loss landscape in facilitating the theoretical comparison of different adaptation options.

Related papers

• MIITA: Memory-Induced Inference-Time Adaptation for Continual Learning with Small Language Models [64.2820121997882]
  MIITA is a Memory-Induced Inference-Time Adaptation framework for supervised CL under constrained storage.<n>It stores supervised experiences as compact correction-direction prototypes with semantic anchors, and retrieves them at inference time using semantic and uncertainty-based cues.<n>Experiments show that MIITA consistently improves final performance and mitigates forgetting under fixed memory budgets.
  arXiv  Detail & Related papers  (2026-05-20T03:03:04Z)
• Pruned Adaptation Modules: A Simple yet Strong Baseline for Continual Foundation Models [9.273551382858525]
  Pruned Adaptation Modules (PAM) is a simple yet effective method that freezes the vast majority of the pre-trained ResNet.<n>PAM consistently mitigates catastrophic forgetting and outperforms state-of-the-art FM-based CIL approaches.
  arXiv  Detail & Related papers  (2026-03-22T11:15:34Z)
• Efficient and Stable Reinforcement Learning for Diffusion Language Models [59.75789436018925]
  Reinforcement Learning (RL) is crucial for unlocking the complex reasoning capabilities of Diffusion-based Large Language Models (dLLMs)<n>Applying to dLLMs faces unique challenges in efficiency and stability.<n>We propose Spatio-Temporal Pruning (STP), a framework designed to simultaneously improve the efficiency and stability of RL for dLLMs.
  arXiv  Detail & Related papers  (2026-02-09T17:04:23Z)
• APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards [61.52322047892064]
  Test-Time Scaling (TTS) has significantly enhanced the capabilities of Large Reasoning Models (LRMs)<n>We observe that LRMs frequently conduct repetitive self-verification without revision even after obtaining the final answer during the reasoning process.<n>We propose Anchor-based Process Reward (APR), a structure-aware reward shaping method that localizes the reasoning anchor and penalizes exclusively the post-anchor AST.
  arXiv  Detail & Related papers  (2026-01-31T14:53:20Z)
• Beyond the Next Port: A Multi-Task Transformer for Forecasting Future Voyage Segment Durations [3.9443085703523706]
  The study reformulates future-port ETA prediction as a segment-level time-series forecasting problem.<n>We develop a transformer-based architecture that integrates historical sailing durations, destination port congestion, and static vessel descriptors.<n>The proposed framework employs a causally masked attention mechanism to capture long-range temporal dependencies and a multi-task learning head to jointly predict segment sailing durations and port congestion states.
  arXiv  Detail & Related papers  (2026-01-12T21:32:05Z)
• Compute-Accuracy Pareto Frontiers for Open-Source Reasoning Large Language Models [0.0]
  Large Language Models (LLMs) are demonstrating rapid improvements on complex reasoning benchmarks.<n>For industrial applications, model selection depends not only on raw accuracy but also on resource constraints and inference costs.
  arXiv  Detail & Related papers  (2025-12-31T10:51:32Z)
• GateTS: Versatile and Efficient Forecasting via Attention-Inspired routed Mixture-of-Experts [0.0]
  We propose a model architecture that simplifies the training process for univariate time series forecasting.<n>Our approach combines sparse MoE computation with a novel attention-inspired gating mechanism that replaces the traditional one-layer softmax router.
  arXiv  Detail & Related papers  (2025-08-24T20:39:50Z)
• WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-training [64.0932926819307]
  We present Warmup-Stable and Merge (WSM), a framework that establishes a formal connection between learning rate decay and model merging.<n>WSM provides a unified theoretical foundation for emulating various decay strategies.<n>Our framework consistently outperforms the widely-adopted Warmup-Stable-Decay (WSD) approach across multiple benchmarks.
  arXiv  Detail & Related papers  (2025-07-23T16:02:06Z)
• FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation [55.12070409045766]
  Post-training quantization (PTQ) has stood out as a cost-effective and promising model compression paradigm in recent years.<n>Current PTQ methods for Vision Transformers (ViTs) still suffer from significant accuracy degradation, especially under low-bit quantization.
  arXiv  Detail & Related papers  (2025-06-13T07:57:38Z)
• PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patches [34.65386386598757]
  PortLLM is a training-free framework that creates an initial lightweight model update patch to capture domain-specific knowledge.<n>PortLLM achieves comparable performance to LoRA fine-tuning with reductions of up to 12.2x in GPU memory usage.
  arXiv  Detail & Related papers  (2024-10-08T13:41:08Z)
• Pareto Low-Rank Adapters: Efficient Multi-Task Learning with Preferences [49.14535254003683]
  We introduce PaLoRA, a novel parameter-efficient method that addresses multi-task trade-offs in machine learning.<n>Our experiments show that PaLoRA outperforms state-of-the-art MTL and PFL baselines across various datasets.
  arXiv  Detail & Related papers  (2024-07-10T21:25:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.