Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization

• URL: http://arxiv.org/abs/2605.17379v1
• Date: Sun, 17 May 2026 10:45:01 GMT
• Title: Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization
• Abstract summary: We introduce a parameter-efficient domain adaptation approach that combines vocabulary adaptation with pretraining for LLM-based text summarization.<n>Our unified framework augments pretrained tokenizers with domain-specific tokens while selectively replacing under-trained and unreachable tokens.<n>We evaluate our approach on Llama-3.1-8B and Qwen2.5-7B across legal and medical summarization tasks.
• Score: 31.777579240285963
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models pretrained on general-domain corpora often exhibit tokenization inefficiencies when applied to specialized domains. Although continual pretraining for domain adaptation partially alleviate performance degradation, it does not resolve the fundamental vocabulary mismatch. To address this gap, we introduce a targeted parameter-efficient domain adaptation approach that combines vocabulary adaptation with pretraining for LLM-based text summarization. Our unified framework augments pretrained tokenizers with domain-specific tokens while selectively replacing under-trained and unreachable tokens to limit parameter growth. We evaluate our approach on Llama-3.1-8B and Qwen2.5-7B across legal and medical summarization tasks on a challenge-oriented evaluation protocol focused on expert-driven text and summaries which typically has higher concentration of over-fragmented Out-of-Vocabulary (OOV) words. The vocabulary adaptation algorithm enhances the overall quality of the summarization model by improving semantic similarity between the generated summaries and their references. In addition, the adapted model produces summaries that incorporate more appropriate novel and domain-specific words, leading to improved coherence, relevance, and faithfulness. We further observe that our proposed approach significantly reduce training time by $35-55\%$ over continual pretraining and reduce parameter counts up to $37\%$ w.r.t expansion-only methods. We make the codebase publicly available at https://github.com/gb-kgp/VocabReplace-Then-Expand.

Related papers

• AdaptBPE: From General Purpose to Specialized Tokenizers [18.70903226766322]
  We propose a post-training adaptation strategy that selectively replaces low-utility tokens with more relevant ones based on their frequency in an adaptation corpus.<n>Our algorithm identifies the token inventory that most effectively encodes the adaptation corpus for a given target vocabulary size.<n>This method serves as a lightweight adaptation mechanism, akin to a vocabulary fine-tuning process, enabling optimized tokenization for specific domains or tasks.
  arXiv  Detail & Related papers  (2026-01-29T12:59:40Z)
• Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Models [12.218365713546214]
  Tokenizer adaptation plays an important role in transferring pre-trained language models to new domains or languages.<n>The common approach to extension trains a new tokenizer on domain-specific text and appends the tokens that do not overlap with the existing vocabulary.<n>We propose continued BPE training, which adapts a pre-trained tokenizer by continuing the BPE merge learning process on new data.
  arXiv  Detail & Related papers  (2025-12-03T17:20:16Z)
• Vocabulary Customization for Efficient Domain-Specific LLM Deployment [7.455386813746299]
  General-domain tokenizer fails to capture frequent domain-specific terms, leading to higher token fertility and thus a decrease in processing speed.<n>We address this limitation by augmenting the pretrained vocabulary with a set of domain-specific tokens.<n> Evaluated on real-world e-Commerce use-cases, the augmented tokenizer significantly shortens input sequences by up to 20%.
  arXiv  Detail & Related papers  (2025-09-30T11:43:12Z)
• Lost in Translation? Vocabulary Alignment for Source-Free Adaptation in Open-Vocabulary Semantic Segmentation [90.5844979560448]
  VocAlign is a source-free domain adaptation framework specifically designed for VLMs in semantic segmentation.<n>Our approach achieves a notable 6.11 mIoU improvement on the CityScapes dataset and demonstrates superior performance on zero-shot segmentation benchmarks.
  arXiv  Detail & Related papers  (2025-09-18T17:59:58Z)
• RecBase: Generative Foundation Model Pretraining for Zero-Shot Recommendation [78.01030342481246]
  RecBase is a domain-agnostic foundational model pretrained with a recommendation-oriented objective.<n>We introduce a unified item tokenizer that encodes items into hierarchical concept identifiers.<n>Our model matches or surpasses the performance of LLM baselines up to 7B parameters in zero-shot and cross-domain recommendation tasks.
  arXiv  Detail & Related papers  (2025-09-03T08:33:43Z)
• HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization [50.27950279695363]
  Many pre-trained language models (PLMs) exhibit suboptimal performance on mid- and low-resource languages.<n>A common strategy to address this is to introduce new tokens specific to the target languages, initialize their embeddings, and apply continual pre-training on target-language data.<n>We propose HYPEROFA, a hypernetwork-based approach for more adaptive token embedding.
  arXiv  Detail & Related papers  (2025-04-21T19:40:32Z)
• Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models [26.442558912559658]
  We show a fundamental limitation in vocabulary adaptation approaches that use Byte-Pair. (BPE) tokenization scheme for fine-tuning pretrained language models (PLMs) to expert domains. We propose AdaptBPE where the BPE tokenization phase is modified to first perform the longest string matching on the added (target) vocabulary before tokenizing at the character level.
  arXiv  Detail & Related papers  (2024-10-04T09:24:55Z)
• Improving Domain Adaptation through Extended-Text Reading Comprehension [108.24709810480654]
  Recent work demonstrates that models using reading comprehension data formatted by adapting-based patterns can significantly improve performance on domain-specific tasks. However, these patterns are incapable of parsing raw corpora using domain-specific knowledge. In comparison to AdaptLLM, our method achieves an improvement exceeding 5% in domain-specific tasks.
  arXiv  Detail & Related papers  (2024-01-14T13:11:31Z)
• Scalable Learning of Latent Language Structure With Logical Offline Cycle Consistency [71.42261918225773]
  Conceptually, LOCCO can be viewed as a form of self-learning where the semantic being trained is used to generate annotations for unlabeled text. As an added bonus, the annotations produced by LOCCO can be trivially repurposed to train a neural text generation model.
  arXiv  Detail & Related papers  (2023-05-31T16:47:20Z)
• Non-Parametric Domain Adaptation for End-to-End Speech Translation [72.37869362559212]
  End-to-End Speech Translation (E2E-ST) has received increasing attention due to the potential of its less error propagation, lower latency, and fewer parameters. We propose a novel non-parametric method that leverages domain-specific text translation corpus to achieve domain adaptation for the E2E-ST system.
  arXiv  Detail & Related papers  (2022-05-23T11:41:02Z)
• Grounded Compositional Outputs for Adaptive Language Modeling [59.02706635250856]
  A language model's vocabulary$-$typically selected before training and permanently fixed later$-$affects its size. We propose a fully compositional output embedding layer for language models. To our knowledge, the result is the first word-level language model with a size that does not depend on the training vocabulary.
  arXiv  Detail & Related papers  (2020-09-24T07:21:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.