MoganColBERT-TR: A Late-Interaction Multi-Vector Retrieval Model for Turkish

• URL: http://arxiv.org/abs/2608.26344v1
• Date: Wed, 26 Aug 2026 19:26:15 GMT
• Title: MoganColBERT-TR: A Late-Interaction Multi-Vector Retrieval Model for Turkish
• Abstract summary: This work introduces the third model in that lineage: MoganColBERT-TR.<n>MoganColBERT-TR is a multi-vector retrieval model that represents it at the token level through a 768->128 projection and scores it with MaxSim late interaction.<n>With 148.9M parameters, MoganColBERT-TR reaches an overall score of 37.36.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We previously reported a ModernBERT encoder trained from scratch for Turkish (MoganBERT-TR) and a single-vector embedding model built on top of it (MoganBERT-embed). This work introduces the third model in that lineage: MoganColBERT-TR, a multi-vector retrieval model that, instead of compressing a query or a document into a single vector, represents it at the token level through a 768->128 projection and scores it with MaxSim late interaction. The model is not trained from scratch: the embedding model's encoder is taken as the starting point and adapted to the ColBERT objective with a single-epoch distillation phase. Training data is produced from two sources - title-to-passage pairs carved out of our own pretraining corpus in the character domain and at sentence boundaries, and two Turkish question-based retrieval sets - and is distilled from the soft scores of a cross-encoder teacher (bge-reranker-v2-m3) over one positive and seven mined negatives. We show that in hard negative mining, rank-based skipping alone is insufficient and must be combined with a group mask and a cosine ceiling. Evaluation is carried out with the official pipeline of TurkColBERT, a benchmark built for Turkish late-interaction retrieval (PLAID index, exact MaxSim), on five Turkish BEIR datasets; none of them appears in our training pool, so all five results are clean zero-shot. With 148.9M parameters, MoganColBERT-TR reaches an overall score of 37.36 (35.53 nDCG@100, 31.81 nDCG@10) averaged over the five datasets and finishes second among the five models compared: it outperforms the twice-as-large ColmmBERT-base-TR on four of five datasets and by +3.05 overall, and the benchmark's largest model by +12.30. The gap to the leading model (mLateOn) is concentrated on ArguAna-TR, the dataset with by far the longest queries.

Related papers

• MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum [0.0]
  MoganBert-TR is trained over 237.3B tokens with a two-stage CLM-to-MLM curriculum.<n>MoganBert-TR attains 78.41 on TrGLUE, the best among the Turkish ModernBERT models compared.
  arXiv  Detail & Related papers  (2026-08-26T13:12:51Z)
• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• Team DACTYL at PAN 2026: Bayesian Data Mixing and Empirical X-risk Minimization for AI-text Detection [0.33842793760651557]
  Existing research shows that AI-generated text detection classifiers achieve strong in-distribution (ID) performance but do not maintain the same performance on out-of-distribution (OOD) texts.<n>We fine-tune BERT-tiny models with Bayesian classification heads to select texts across three different datasets to use as a consolidated training set.<n>Our results highlight that careful dataset curation can lead to strong OOD performance.
  arXiv  Detail & Related papers  (2026-07-19T18:48:59Z)
• TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish [0.7233065479782755]
  TabiBERT is a monolingual Turkish encoder based on ModernBERT architecture trained from scratch on a large, curated corpus.<n>It supports 8,192-token context length (16x original BERT), achieves up to 2.65x speedup, and reduces GPU memory consumption.<n>It attains 77.58 on TabiBench, outperforming BERTurk by 1.62 points and establishing state-of-the-art on five of eight categories.
  arXiv  Detail & Related papers  (2025-12-28T20:18:22Z)
• TurkColBERT: A Benchmark of Dense and Late-Interaction Models for Turkish Information Retrieval [0.0]
  We introduce TurkColBERT, the first benchmark comparing dense encoders and late-interaction models for Turkish retrieval.<n>Our two-stage adaptation pipeline fine-tunes English and multilingual encoders on Turkish NLI/STS tasks, then converts them into ColBERT-style retrievers.<n>We evaluate 10 models across five Turkish BEIR datasets covering scientific, financial, and argumentative domains.
  arXiv  Detail & Related papers  (2025-11-20T16:42:21Z)
• MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes [60.57770396565211]
  We show that strong reasoning abilities can emerge with far less data.<n>MobileLLM-R50M achieves an AIME score of 15.5, compared to just 0.6 for OLMo-2-1.48B and 0.3 for SmolLM-2-1.7B.
  arXiv  Detail & Related papers  (2025-09-29T15:43:59Z)
• mmBERT: A Modern Multilingual Encoder with Annealed Language Learning [57.58071656545661]
  mmBERT is an encoder-only language model pretrained on 3T tokens of multilingual text.<n>We add over 1700 low-resource languages to the data mix only during the decay phase.<n>We show that mmBERT significantly outperforms the previous generation of models on classification and retrieval tasks.
  arXiv  Detail & Related papers  (2025-09-08T17:08:42Z)
• MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm [60.14048367611333]
  MonkeyOCR is a vision-language model for document parsing.<n>It advances the state of the art by leveraging a Structure-Recognition-Relation (SRR) triplet paradigm.
  arXiv  Detail & Related papers  (2025-06-05T16:34:57Z)
• Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval [49.1574468325115]
  We introduce Amharic-specific dense retrieval models based on pre-trained Amharic BERT and RoBERTa backbones.<n>Our proposed RoBERTa-Base-Amharic-Embed model (110M parameters) achieves a 17.6% relative improvement in MRR@10.<n>More compact variants, such as RoBERTa-Medium-Amharic-Embed (42M) remain competitive while being over 13x smaller.
  arXiv  Detail & Related papers  (2025-05-25T23:06:20Z)
• Data-Efficient French Language Modeling with CamemBERTa [0.0]
  We introduce CamemBERTa, a French DeBERTa model that builds upon the DeBERTaV3 architecture and training objective. We evaluate our model's performance on a variety of French downstream tasks and datasets.
  arXiv  Detail & Related papers  (2023-06-02T12:45:34Z)
• Evaluation of Transfer Learning for Polish with a Text-to-Text Model [54.81823151748415]
  We introduce a new benchmark for assessing the quality of text-to-text models for Polish. The benchmark consists of diverse tasks and datasets: KLEJ benchmark adapted for text-to-text, en-pl translation, summarization, and question answering. We present plT5 - a general-purpose text-to-text model for Polish that can be fine-tuned on various Natural Language Processing (NLP) tasks with a single training objective.
  arXiv  Detail & Related papers  (2022-05-18T09:17:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.