moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT

• URL: http://arxiv.org/abs/2606.22722v1
• Date: Sun, 21 Jun 2026 23:44:47 GMT
• Title: moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT
• Abstract summary: moBERTo is a Portuguese adaptation of ModernBERT obtained through continued pretraining on 60 billion tokens.<n>We preserve the original architecture, including rotary positional embeddings, alternating local-global attention, flash attention, and unpadding.<n>Our best variant, which combines a Portuguese tokenizer with subword-matching embedding transfer and long-context post-training, achieves the highest average reranking nDCG@10.
• Score: 4.563830993050022
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Encoder-only transformer models remain essential for production NLP pipelines. We introduce moBERTo, a Portuguese adaptation of ModernBERT obtained through continued pretraining of the ModernBERT-base checkpoint on 60 billion tokens (5 epochs over a 12-billion-token corpus curated from FineWeb2 and filtered with educational and STEM classifiers). We preserve the original architecture, including rotary positional embeddings, alternating local-global attention, flash attention, and unpadding. We evaluate moBERTo across information retrieval (including long-context retrieval at up to 8,192 tokens), document classification, named entity recognition, and natural language understanding. Our best variant, which combines a Portuguese tokenizer with subword-matching embedding transfer and long-context post-training, achieves the highest average reranking nDCG@10 across three Portuguese retrieval benchmarks and the best results on PLUE-PT. Through ablation studies, we show that (i) continued pretraining is strongly preferable to training from scratch, particularly for preserving long-context capabilities; (ii) tokenizer adaptation improves token-level tasks but degrades long-context retrieval; (iii) a dedicated long-context post-training phase at 8,192 tokens further improves reranking and NER; and (iv) encoder-only architectures remain competitive with larger decoder-only alternatives for discriminative tasks. We publicly release the model weights at https://huggingface.co/Tropic-AI/moBERTo and training data at https://huggingface.co/datasets/Tropic-AI/moberto-pretraining-dataset-c4-compatible on Hugging Face.

Related papers

• NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction [76.38136259305638]
  We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP)<n> NCP-ArchPreview builds a latent space by constructing a product-quantized concept vocabulary directly from its hidden states, and subsequently learns to predict future concepts via a dedicated Concept Module.<n>We scale this architecture to 8.9B parameters and train it on 5.73T tokens from the Dolma-3 dataset, marking the largest demonstration of a latent-space language model to date.
  arXiv  Detail & Related papers  (2026-09-09T18:12:43Z)
• REER-PT: Reverse-Engineered Reasoning for Perplexity-Guided Pre-training Data Augmentation [64.84437332310516]
  We introduce textbfREER-PT, a scalable framework that extends Reverse-Engineered Reasoning (REER) to raw pre-training data.<n>REER-PT identifies continuations that are difficult to predict but can still be inferred from the preceding context, and inserts concise reasoning annotations that reconstruct the missing connection between context and continuation.<n>This sparse transformation preserves the source text and remains compatible with standard next-token prediction, avoiding online reasoning rollouts during pre-training.
  arXiv  Detail & Related papers  (2026-08-31T11:34:19Z)
• TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish [0.7233065479782755]
  TabiBERT is a monolingual Turkish encoder based on ModernBERT architecture trained from scratch on a large, curated corpus.<n>It supports 8,192-token context length (16x original BERT), achieves up to 2.65x speedup, and reduces GPU memory consumption.<n>It attains 77.58 on TabiBench, outperforming BERTurk by 1.62 points and establishing state-of-the-art on five of eight categories.
  arXiv  Detail & Related papers  (2025-12-28T20:18:22Z)
• SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptation [40.55805997909858]
  We introduce SpidR-Adapt for rapid adaptation to new languages using minimal unlabeled data.<n>We construct a multi-task adaptive pre-training protocol which formulates the adaptation process as a bi-level optimization framework.<n> Empirically, SpidR-Adapt achieves rapid gains in phonemic discriminability and spoken language modeling.
  arXiv  Detail & Related papers  (2025-12-24T14:33:16Z)
• llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length [1.5857828218932415]
  We present llm-jp-modernbert, a ModernBERT model trained on a publicly available, massive Japanese corpus with a context length of 8192 tokens.<n>While our model does not surpass existing baselines on downstream tasks, it achieves good results on fill-mask test evaluations.
  arXiv  Detail & Related papers  (2025-04-22T02:45:19Z)
• MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining [10.421048804389343]
  We introduce MosaicBERT, a BERT-style encoder architecture and training recipe that is empirically optimized for fast pretraining. When pretrained from scratch on the C4 dataset, this base model achieves a downstream average GLUE (dev) score of 79.6 in 1.13 hours on 8 A100 80 GB GPUs at a cost of roughly $20. This empirical speed up in pretraining enables researchers and engineers to pretrain custom BERT-style models at low cost instead of finetune on existing generic models.
  arXiv  Detail & Related papers  (2023-12-29T06:05:19Z)
• Strategies for improving low resource speech to text translation relying on pre-trained ASR models [59.90106959717875]
  This paper presents techniques and findings for improving the performance of low-resource speech to text translation (ST) We conducted experiments on both simulated and real-low resource setups, on language pairs English - Portuguese, and Tamasheq - French respectively.
  arXiv  Detail & Related papers  (2023-05-31T21:58:07Z)
• CTC-based Non-autoregressive Speech Translation [51.37920141751813]
  We investigate the potential of connectionist temporal classification for non-autoregressive speech translation. We develop a model consisting of two encoders that are guided by CTC to predict the source and target texts. Experiments on the MuST-C benchmarks show that our NAST model achieves an average BLEU score of 29.5 with a speed-up of 5.67$times$.
  arXiv  Detail & Related papers  (2023-05-27T03:54:09Z)
• NarrowBERT: Accelerating Masked Language Model Pretraining and Inference [50.59811343945605]
  We propose NarrowBERT, a modified transformer encoder that increases the throughput for masked language model pretraining by more than $2times$. NarrowBERT sparsifies the transformer model such that the self-attention queries and feedforward layers only operate on the masked tokens of each sentence during pretraining. We show that NarrowBERT increases the throughput at inference time by as much as $3.5times$ with minimal (or no) performance degradation on sentence encoding tasks like MNLI.
  arXiv  Detail & Related papers  (2023-01-11T23:45:50Z)
• Exploring Unsupervised Pretraining Objectives for Machine Translation [99.5441395624651]
  Unsupervised cross-lingual pretraining has achieved strong results in neural machine translation (NMT) Most approaches adapt masked-language modeling (MLM) to sequence-to-sequence architectures, by masking parts of the input and reconstructing them in the decoder. We compare masking with alternative objectives that produce inputs resembling real (full) sentences, by reordering and replacing words based on their context.
  arXiv  Detail & Related papers  (2021-06-10T10:18:23Z)
• From Universal Language Model to Downstream Task: Improving RoBERTa-Based Vietnamese Hate Speech Detection [8.602181445598776]
  We propose a pipeline to adapt the general-purpose RoBERTa language model to a specific text classification task: Vietnamese Hate Speech Detection. Our experiments proved that our proposed pipeline boosts the performance significantly, achieving a new state-of-the-art on Vietnamese Hate Speech Detection campaign with 0.7221 F1 score.
  arXiv  Detail & Related papers  (2021-02-24T09:30:55Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.