Optimizing Korean-Centric LLMs via Token Pruning

• URL: http://arxiv.org/abs/2604.16235v1
• Date: Fri, 17 Apr 2026 16:53:14 GMT
• Title: Optimizing Korean-Centric LLMs via Token Pruning
• Abstract summary: token pruning is a compression technique that eliminates tokens and embedding parameters corresponding to languages irrelevant to the target application.<n>We evaluate architectures including Qwen3, Gemma-3, Llama-3, and Aya across three vocabulary configurations.
• Score: 6.029880646740327
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents a systematic benchmark of state-of-the-art multilingual large language models (LLMs) adapted via token pruning - a compression technique that eliminates tokens and embedding parameters corresponding to languages irrelevant to the target application. Focusing on Korean-centric natural language processing (NLP) tasks, we evaluate architectures including Qwen3, Gemma-3, Llama-3, and Aya across three vocabulary configurations: Original, English-Korean (EnKo), and English-Korean-Chinese (EnKoZh). Performance is assessed using established benchmarks for general aptitude, cultural literacy, instruction following, and machine translation. Our findings indicate that token pruning significantly improves generation stability by eliminating language confusion, and in the case of machine translation, frequently enhances performance on Korean-specific tasks. While instruction-following capabilities display architecture-dependent variance linked to latent cross-lingual representations, the significant reduction in vocabulary size validates token pruning as a highly effective optimization strategy for memory-constrained, domain-specific deployments, despite modest gains in inference latency.

Related papers

• Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models [30.7730386396743]
  Byte-level Byte-Pair tokenizers structurally favor high-resource languages and Latin scripts.<n>For speakers of underrepresented languages, this bias inflates inference costs and widens cross-lingual capability gaps.<n>We present the first systematic comparison of equitable tokenizers on a benchmark spanning 11 Southeast Asian languages.
  arXiv  Detail & Related papers  (2026-06-13T01:10:42Z)
• What Language is This? Ask Your Tokenizer [32.28976119949841]
  Language Identification (LID) is an important component of many multilingual natural language processing pipelines.<n>We introduce UniLID, a simple and efficient LID method based on the UnigramLM tokenization algorithm.<n>Our formulation is data- and compute-efficient, supports incremental addition of new languages without retraining existing models.
  arXiv  Detail & Related papers  (2026-02-19T18:58:39Z)
• Finding the Translation Switch: Discovering and Exploiting the Task-Initiation Features in LLMs [69.28193153685893]
  Large Language Models (LLMs) frequently exhibit strong translation abilities, even without task-specific fine-tuning.<n>To demystify this process, we leverage Sparse Autoencoders (SAEs) and introduce a novel framework for identifying task-specific features.<n>Our work not only decodes a core component of the translation mechanism in LLMs but also provides a blueprint for using internal model mechanism to create more robust and efficient models.
  arXiv  Detail & Related papers  (2026-01-16T06:29:07Z)
• Edit Once, Update Everywhere: A Simple Framework for Cross-Lingual Knowledge Synchronization in LLMs [60.12222055772508]
  We present a simple and practical state-of-the-art (SOTA) recipe Cross-Lingual Knowledge Democracy Edit (X-KDE)<n>X-KDE is designed to propagate knowledge from a dominant language to other languages effectively.<n>Experiments on the Bi-ZsRE and MzsRE benchmarks show that X-KDE significantly enhances cross-lingual performance.
  arXiv  Detail & Related papers  (2025-02-20T15:32:31Z)
• Language Fusion for Parameter-Efficient Cross-lingual Transfer [21.96231169571248]
  Fusion forLanguage Representations (FLARE) is a novel method that enhances representation quality and downstream performance for languages other than English.<n>FLARE integrates source and target language representations within low-rank (LoRA) adapters using lightweight linear transformations.<n>A series of experiments across representative cross-lingual natural language understanding tasks, including natural language inference, question-answering and sentiment analysis, demonstrate FLARE's effectiveness.
  arXiv  Detail & Related papers  (2025-01-12T18:02:29Z)
• SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator [65.62084602011596]
  Large Language Models (LLMs) have exhibited exceptional performance across a spectrum of natural language processing tasks.<n>We have identified a key pattern: certain seemingly meaningless separator tokens (i.e., punctuations) contribute disproportionately to attention scores compared to semantically meaningful tokens.<n>We introduce SepLLM, a plug-and-play framework that accelerates inference by compressing these segments and eliminating redundant tokens.
  arXiv  Detail & Related papers  (2024-12-16T18:58:57Z)
• When Every Token Counts: Optimal Segmentation for Low-Resource Language Models [0.0]
  We show that an optimal Byte-Pair (BPE) configuration significantly reduces token count compared to greedy segmentation.<n>Our findings suggest that compression-optimized tokenization strategies could provide substantial advantages for multilingual and low-resource language applications.
  arXiv  Detail & Related papers  (2024-12-09T19:11:54Z)
• Bridging the Language Gaps in Large Language Models with Inference-Time Cross-Lingual Intervention [71.12193680015622]
  Large Language Models (LLMs) have shown remarkable capabilities in natural language processing. LLMs exhibit significant performance gaps among different languages. We propose Inference-Time Cross-Lingual Intervention (INCLINE) to overcome these limitations without incurring significant costs.
  arXiv  Detail & Related papers  (2024-10-16T11:23:03Z)
• Improving Korean NLP Tasks with Linguistically Informed Subword Tokenization and Sub-character Decomposition [6.767341847275751]
  We introduce a morpheme-aware subword tokenization method that utilizes sub-character decomposition to address the challenges of applying Byte Pair. Our approach balances linguistic accuracy with computational efficiency in Pre-trained Language Models (PLMs) Our evaluations show that this technique achieves good performances overall, notably improving results in the syntactic task of NIKL-CoLA.
  arXiv  Detail & Related papers  (2023-11-07T12:08:21Z)
• GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extraction [107.8262586956778]
  We introduce graph convolutional networks (GCNs) with universal dependency parses to learn language-agnostic sentence representations. GCNs struggle to model words with long-range dependencies or are not directly connected in the dependency tree. We propose to utilize the self-attention mechanism to learn the dependencies between words with different syntactic distances.
  arXiv  Detail & Related papers  (2020-10-06T20:30:35Z)
• Self-Attention with Cross-Lingual Position Representation [112.05807284056337]
  Position encoding (PE) is used to preserve the word order information for natural language processing tasks, generating fixed position indices for input sequences. Due to word order divergences in different languages, modeling the cross-lingual positional relationships might help SANs tackle this problem. We augment SANs with emphcross-lingual position representations to model the bilingually aware latent structure for the input sentence.
  arXiv  Detail & Related papers  (2020-04-28T05:23:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.