Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings

• URL: http://arxiv.org/abs/2512.12167v1
• Date: Sat, 13 Dec 2025 04:23:47 GMT
• Title: Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
• Abstract summary: Dropping the Positional Embeddings of LMs after training (DroPE) is motivated by three key theoretical and empirical observations.<n> positional embeddings (PEs) serve a crucial role during pretraining, providing an important inductive bias.<n>DroPE yields seamless zero-shot context extension without any long-context finetuning.
• Score: 14.342802193320365
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: So far, expensive finetuning beyond the pretraining sequence length has been a requirement for effectively extending the context of language models (LM). In this work, we break this key bottleneck by Dropping the Positional Embeddings of LMs after training (DroPE). Our simple method is motivated by three key theoretical and empirical observations. First, positional embeddings (PEs) serve a crucial role during pretraining, providing an important inductive bias that significantly facilitates convergence. Second, over-reliance on this explicit positional information is also precisely what prevents test-time generalization to sequences of unseen length, even when using popular PE-scaling methods. Third, positional embeddings are not an inherent requirement of effective language modeling and can be safely removed after pretraining, following a short recalibration phase. Empirically, DroPE yields seamless zero-shot context extension without any long-context finetuning, quickly adapting pretrained LMs without compromising their capabilities in the original training context. Our findings hold across different models and dataset sizes, far outperforming previous specialized architectures and established rotary positional embedding scaling methods.

Related papers

• Representation Collapse in Sequential Post-Training of Large Language Models [17.437127662981847]
  We study whether sequential post-training gradually compresses internal representations into low-rank, anisotropic, and homogeneous feature spaces.<n>We analyze supervised fine-tuning, preference optimization, safety/refusal tuning, math and code specialization, and long chain-of-thought tuning under controlled stage orderings.
  arXiv  Detail & Related papers  (2026-05-28T19:59:17Z)
• Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP [0.0]
  Subword regularization methods such as BPE dropout are applied only during fine-tuning, while pretraining is usually done with deterministic tokenization.<n>This creates a potential segmentation mismatch between pretraining and fine-tuning.<n>We train monolingual and bilingual BERT models on downsampled subsets of English, German, French, Spanish, Kiswahili, and isiXhosa, and evaluate them on XNLI, PAWS-X, PAN-X, and MasakhaNER 2.0.
  arXiv  Detail & Related papers  (2026-05-13T12:31:04Z)
• Forward-Only Continual Learning [8.873948519614244]
  Catastrophic forgetting remains a central challenge in continual learning.<n>We propose FoRo, a forward-only, gradient-free continual learning method.<n>Experiments show that FoRo significantly reduces average forgetting and improves accuracy.
  arXiv  Detail & Related papers  (2025-09-01T15:10:38Z)
• Steering Information Utility in Key-Value Memory for Language Model Post-Training [16.655945393684373]
  We introduce InfoSteer, a lightweight method that encourages parametric information utilization in language models (LMs) during post-training.<n>This simple guidance yields consistent performance improvements across 15 downstream tasks in both in-distribution (ID) and out-of-distribution (OOD) evaluations.<n>Our work underscores that vanilla post-training does not fully exploit the potential gained during pre-training, and that steering LMs in latent representation space offers a promising approach to enhance both performance and interpretability.
  arXiv  Detail & Related papers  (2025-07-07T16:13:21Z)
• Long-Tailed Object Detection Pre-training: Dynamic Rebalancing Contrastive Learning with Dual Reconstruction [28.359463356384463]
  We introduce a novel pre-training framework for object detection, called Dynamic Rebalancing Contrastive Learning with Dual Reconstruction (2DRCL) Our method builds on a Holistic-Local Contrastive Learning mechanism, which aligns pre-training with object detection by capturing both global contextual semantics and detailed local patterns. Experiments on COCO and LVIS v1.0 datasets demonstrate the effectiveness of our method, particularly in improving the mAP/AP scores for tail classes.
  arXiv  Detail & Related papers  (2024-11-14T13:59:01Z)
• Accelerating Large Language Model Inference with Self-Supervised Early Exits [0.0]
  This paper presents a novel technique for accelerating inference in large, pre-trained language models (LLMs) We propose the integration of early exit ''heads'' atop existing transformer layers, which facilitate conditional terminations based on a confidence metric.
  arXiv  Detail & Related papers  (2024-07-30T07:58:28Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis [27.310894780313618]
  This paper undertakes a comprehensive comparison of model capabilities at various pretraining intermediate checkpoints. We confirm that specific downstream metrics exhibit similar training dynamics across models of different sizes. In addition to our core findings, we've reproduced Amber and OpenLLaMA, releasing their intermediate checkpoints.
  arXiv  Detail & Related papers  (2024-04-01T16:00:01Z)
• Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition [54.9235160379917]
  Stable Distillation is a simple and novel approach for SSL-based continued pre-training. It boosts ASR performance in the target domain where both labeled and unlabeled data are limited.
  arXiv  Detail & Related papers  (2023-12-20T06:02:12Z)
• Initialization Matters for Adversarial Transfer Learning [61.89451332757625]
  We discover the necessity of an adversarially robust pretrained model. We propose Robust Linear Initialization (RoLI) for adversarial finetuning, which initializes the linear head with the weights obtained by adversarial linear probing. Across five different image classification datasets, we demonstrate the effectiveness of RoLI and achieve new state-of-the-art results.
  arXiv  Detail & Related papers  (2023-12-10T00:51:05Z)
• How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? [92.90857135952231]
  Transformers pretrained on diverse tasks exhibit remarkable in-context learning (ICL) capabilities. We study ICL in one of its simplest setups: pretraining a linearly parameterized single-layer linear attention model for linear regression.
  arXiv  Detail & Related papers  (2023-10-12T15:01:43Z)
• Adversarial Self-Attention for Language Understanding [89.265747130584]
  This paper proposes textitAdversarial Self-Attention mechanism (ASA). ASA adversarially reconstructs the Transformer attentions and facilitates model training from contaminated model structures. For fine-tuning, ASA-empowered models consistently outweigh naive models by a large margin considering both generalization and robustness.
  arXiv  Detail & Related papers  (2022-06-25T09:18:10Z)
• On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets [74.11825654535895]
  Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance. We study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks.
  arXiv  Detail & Related papers  (2021-09-08T10:39:57Z)
• Back-Translated Task Adaptive Pretraining: Improving Accuracy and Robustness on Text Classification [5.420446976940825]
  We propose a back-translated task-adaptive pretraining (BT-TAPT) method that increases the amount of task-specific data for LM re-pretraining. The experimental results show that the proposed BT-TAPT yields improved classification accuracy on both low- and high-resource data and better robustness to noise than the conventional adaptive pretraining method.
  arXiv  Detail & Related papers  (2021-07-22T06:27:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.