Long-Context Encoder Models for Polish Language Understanding

• URL: http://arxiv.org/abs/2603.12191v1
• Date: Thu, 12 Mar 2026 17:21:45 GMT
• Title: Long-Context Encoder Models for Polish Language Understanding
• Abstract summary: High-quality Polish model capable of processing sequences of up to 8192 tokens.<n>Model was developed by employing a two-stage training procedure that involves positional embedding adaptation and full parameter continuous pre-training.<n>Results demonstrate that our model achieves the best average performance among Polish and multilingual models.
• Score: 0.015871271027849004
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While decoder-only Large Language Models (LLMs) have recently dominated the NLP landscape, encoder-only architectures remain a cost-effective and parameter-efficient standard for discriminative tasks. However, classic encoders like BERT are limited by a short context window, which is insufficient for processing long documents. In this paper, we address this limitation for the Polish by introducing a high-quality Polish model capable of processing sequences of up to 8192 tokens. The model was developed by employing a two-stage training procedure that involves positional embedding adaptation and full parameter continuous pre-training. Furthermore, we propose compressed model variants trained via knowledge distillation. The models were evaluated on 25 tasks, including the KLEJ benchmark, a newly introduced financial task suite (FinBench), and other classification and regression tasks, specifically those requiring long-document understanding. The results demonstrate that our model achieves the best average performance among Polish and multilingual models, significantly outperforming competitive solutions in long-context tasks while maintaining comparable quality on short texts.

Related papers

• Pretraining and Benchmarking Modern Encoders for Latvian [0.16752182911522517]
  Low-resource languages such as Latvian remain underrepresented in pretraining corpora.<n>We address this gap by pretraining a suite of Latvian-specific encoders based on RoBERTa, DeBERTaV3, and ModernBERT architectures.<n>Our models are competitive with existing monolingual and multilingual encoders while benefiting from recent architectural and efficiency advances.
  arXiv  Detail & Related papers  (2026-03-16T09:10:55Z)
• Stacked from One: Multi-Scale Self-Injection for Context Window Extension [69.24689919827817]
  modelname is a novel framework based on multi-grained context compression and query-aware information acquisition.<n>modelnameachieves performance superior or comparable to strong baselines.
  arXiv  Detail & Related papers  (2026-03-05T03:16:16Z)
• Language Ranker: A Lightweight Ranking framework for LLM Decoding [70.01564145836129]
  This paper conceptualizes the decoding process as analogous to the ranking stage in recommendation pipelines.<n>Motivated by this insight, we propose Language Ranker, a novel framework that introduces a lightweight module to rerank candidate responses.<n> Experiments show that Language Ranker achieves performance comparable to large-scale reward models, while requiring only 0.5M additional parameters.
  arXiv  Detail & Related papers  (2025-10-23T17:56:46Z)
• Investigating the translation capabilities of Large Language Models trained on parallel data only [1.5974665548135587]
  Large Language Models (LLMs) have demonstrated exceptional proficiency across a broad spectrum of Natural Language Processing (NLP) tasks. We introduce PLUME, a collection of three 2B LLMs featuring varying vocabulary sizes (32k, 128k, and 256k) trained exclusively on Catalan-centric parallel examples. These models perform comparably to previous encoder-decoder architectures on 16 supervised translation directions and 56 zero-shot ones.
  arXiv  Detail & Related papers  (2024-06-13T14:08:56Z)
• Simultaneous Machine Translation with Large Language Models [51.470478122113356]
  We investigate the possibility of applying Large Language Models to SimulMT tasks. We conducted experiments using the textttLlama2-7b-chat model on nine different languages from the MUST-C dataset. The results show that LLM outperforms dedicated MT models in terms of BLEU and LAAL metrics.
  arXiv  Detail & Related papers  (2023-09-13T04:06:47Z)
• A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models [87.7086269902562]
  We show that subword-based models might still be the most practical choice in many settings. We encourage future work in tokenizer-free methods to consider these factors when designing and evaluating new models.
  arXiv  Detail & Related papers  (2022-10-13T15:47:09Z)
• MuLD: The Multitask Long Document Benchmark [4.835289158553091]
  We present a new long document benchmark consisting of only documents over 10,000 tokens. We show that models with increased context length are better able to solve the tasks presented.
  arXiv  Detail & Related papers  (2022-02-15T12:42:55Z)
• CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain [22.846469609263416]
  We introduce the pre-trained CLIN-X (Clinical XLM-R) language models and show how CLIN-X outperforms other pre-trained transformer models. Our studies reveal stable model performance despite a lack of annotated data with improvements of up to 47 F1 points when only 250 labeled sentences are available. Our results highlight the importance of specialized language models as CLIN-X for concept extraction in non-standard domains.
  arXiv  Detail & Related papers  (2021-12-16T10:07:39Z)
• LegaLMFiT: Efficient Short Legal Text Classification with LSTM Language Model Pre-Training [0.0]
  Large Transformer-based language models such as BERT have led to broad performance improvements on many NLP tasks. In legal NLP, BERT-based models have led to new state-of-the-art results on multiple tasks. We show that lightweight LSTM-based Language Models are able to capture enough information from a small legal text pretraining corpus and achieve excellent performance on short legal text classification tasks.
  arXiv  Detail & Related papers  (2021-09-02T14:45:04Z)
• LOT: A Benchmark for Evaluating Chinese Long Text Understanding and Generation [49.57366550980932]
  Long text modeling requires many capabilities such as modeling long-range commonsense and discourse relations. We propose LOT, a benchmark including two understanding and two generation tasks for Chinese long text modeling evaluation. We release an encoder-decoder Chinese long text pretraining model named LongLM with up to 1 billion parameters.
  arXiv  Detail & Related papers  (2021-08-30T02:38:32Z)
• MOROCCO: Model Resource Comparison Framework [61.444083353087294]
  We present MOROCCO, a framework to compare language models compatible with ttjiant environment which supports over 50 NLU tasks. We demonstrate its applicability for two GLUE-like suites in different languages.
  arXiv  Detail & Related papers  (2021-04-29T13:01:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.