Chinese ModernBERT with Whole-Word Masking

• URL: http://arxiv.org/abs/2510.12285v1
• Date: Tue, 14 Oct 2025 08:41:22 GMT
• Title: Chinese ModernBERT with Whole-Word Masking
• Abstract summary: We introduce Chinese ModernBERT, a hardware-aware 32k BPE vocabulary tailored to frequent Chinese affixes/compounds.<n>We pre-train on 1.2T Chinese tokens from CCI3-HQ, CCI4 (Chinese), and Cosmopedia-Chinese.<n>On CLUE, Chinese ModernBERT is competitive with strong Chinese encoders under a unified fine-tuning protocol.
• Score: 15.633276132474316
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Encoder-only Transformers have advanced along three axes -- architecture, data, and systems -- yielding Pareto gains in accuracy, speed, and memory efficiency. Yet these improvements have not fully transferred to Chinese, where tokenization and morphology differ markedly from English. We introduce Chinese ModernBERT, a from-scratch Chinese encoder that couples: (i) a hardware-aware 32k BPE vocabulary tailored to frequent Chinese affixes/compounds, lowering the embedding budget; (ii) whole-word masking (WWM) with a dynamic masking curriculum (30% -> 15%) to align task difficulty with training progress; (iii) a two-stage pre-training pipeline that extends the native context from 1,024 to 8,192 tokens using RoPE and alternating local/global attention; and (iv) a damped-cosine learning-rate schedule for stable long-horizon optimization. We pre-train on ~1.2T Chinese tokens from CCI3-HQ, CCI4 (Chinese), and Cosmopedia-Chinese. On CLUE, Chinese ModernBERT is competitive with strong Chinese encoders under a unified fine-tuning protocol. Under bf16 it achieves high long-sequence throughput while maintaining strong short-sequence speed, reflecting benefits from budget allocation and attention design. To probe retrieval-oriented quality, we add a small amount of open contrastive data: fine-tuning on SimCLUE (~3M pairs) improves further when adding T2Ranking (~2M), reaching 0.505 (Pearson) / 0.537 (Spearman) on the SimCLUE test set. Under this open-data setting, Chinese ModernBERT surpasses Qwen-0.6B-embedding on SimCLUE, suggesting a clear scaling path for STS with additional curated pairs. We will release tokenizer and weights to facilitate reproducible research.

Related papers

• CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment [55.74660714572696]
  Malicious content generated from large language models (LLMs) could pose severe safety risks and ethical concerns.<n>Existing LLM safety guardrails excel in English or multilingual settings, but lack adaptation to Chinese-specific regulatory policies, cultural context and linguistic nuances.<n>We introduce a 5-macro, 31-micro category fine-grained risk taxonomy for Chinese scenarios, and build CHILLGuard: a dedicated Chinese LLM content safety guardrail.
  arXiv  Detail & Related papers  (2026-06-13T16:57:51Z)
• Reference-Free Reinforcement Learning Fine-Tuning for MT: A Seq2Seq Perspective [1.786053901581251]
  GRPO yields consistent improvements on all 13 languages, up to $+$5.03 chrF++ for Traditional Chinese, and, without any target-language data, competes with 3-epoch supervised fine-tuning on morphologically complex languages.<n>We identify a consistent empirical pattern in which gains are largest where baseline performance is weakest and reward discriminability is highest.
  arXiv  Detail & Related papers  (2026-05-15T14:11:23Z)
• MIPIAD: Multilingual Indirect Prompt Injection Attack Defense with Qwen -- TF-IDF Hybrid and Meta-Ensemble Learning [0.7161783472741748]
  MIPIAD is a defense framework evaluated on English and Bangla.<n>It combines a sequence fine-tuned from Qwen2.5-1.5B via LoRA (XLPID), TF-IDF lexical features, and validation-tuned ensembling.
  arXiv  Detail & Related papers  (2026-05-08T05:34:28Z)
• A Reproducible Multi-Architecture Baseline for Token-Level Chinese Metaphor Identification under the MIPVU Framework [2.545461559283292]
  This paper presents a reproducible multi-architecture baseline for token-level metaphor identification on the PSU Chinese Metaphor Corpus (PSU CMC)<n>We compare three model families: encoder fine-tuning with Chinese RoBERTa-wwm-ext-large; (ii) MelBERT adapted to Chinese using a newly constructed basic-meaning resource derived from the Modern Chinese Dictionary, 7th edition (MCD7)<n>Across five fixed seeds, MelBERT MIP-only achieves the strongest performance at 0.7281 +/- 0.0050 test positive F1, marginally above MelBERT Full (0.7270 +/- 0.0069) and clearly above plain RoBERTa (0.7
  arXiv  Detail & Related papers  (2026-05-08T03:07:19Z)
• Qwen3-TTS Technical Report [64.94647392030824]
  We present the Qwen3-TTS series, a family of advanced multilingual, controllable, robust, and streaming text-to-speech models.<n>Qwen3-TTS supports state-of-the-art 3-second voice cloning and description-based control.<n>Qwen3-TTS adopts a dual-track LM architecture for real-time synthesis, coupled with two speech tokenizers.
  arXiv  Detail & Related papers  (2026-01-22T03:51:43Z)
• Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPO [0.0]
  Small Language Models (SLMs) enable cost-effective, on-device and latency-sensitive AI applications.<n>Yet their deployment in Traditional Chinese (TC) remains hindered by token-level instability.<n>We address this practical reliability gap by creating PureTC-1B, a three-stage stabilization pipeline for Llama-3.2-1B-Instruct.
  arXiv  Detail & Related papers  (2025-10-02T02:50:12Z)
• COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for Alignment with Human Values [43.09443095372083]
  We introduce COIG-P (Chinese Open Instruction Generalist - Preference), a high-quality, large-scale Chinese preference dataset.<n>It comprises 1,009k Chinese preference pairs spanning 6 diverse domains: Chat, Code, Math, Logic, Novel, and Role.<n>Building upon COIG-P, to reduce the overhead of using LLMs for scoring, we trained a 8B-sized Chinese Reward Model.
  arXiv  Detail & Related papers  (2025-04-07T22:15:51Z)
• COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning [37.843051974342124]
  We introduce COIG-CQIA, a new Chinese instruction tuning dataset derived from various real-world resources and undergoing rigorous human verification. We conduct extensive experiments on COIG-CQIA, and compare them with strong baseline models and datasets. The experimental results show that models trained on COIG-CQIA achieve highly competitive performance in diverse benchmarks.
  arXiv  Detail & Related papers  (2024-03-26T19:24:18Z)
• CTC-based Non-autoregressive Speech Translation [51.37920141751813]
  We investigate the potential of connectionist temporal classification for non-autoregressive speech translation. We develop a model consisting of two encoders that are guided by CTC to predict the source and target texts. Experiments on the MuST-C benchmarks show that our NAST model achieves an average BLEU score of 29.5 with a speed-up of 5.67$times$.
  arXiv  Detail & Related papers  (2023-05-27T03:54:09Z)
• Efficient Cross-Lingual Transfer for Chinese Stable Diffusion with Images as Pivots [80.32906566894171]
  We propose IAP, a simple but effective method to transfer English Stable Diffusion into Chinese. IAP establishes connections of Chinese, English and visual semantics in CLIP's embedding space efficiently. Experimental results show that our method outperforms several strong Chinese diffusion models with only 5%10% training data.
  arXiv  Detail & Related papers  (2023-05-19T09:20:27Z)
• M3ST: Mix at Three Levels for Speech Translation [66.71994367650461]
  We propose Mix at three levels for Speech Translation (M3ST) method to increase the diversity of the augmented training corpus. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. Experiments on MuST-C speech translation benchmark and analysis show that M3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.
  arXiv  Detail & Related papers  (2022-12-07T14:22:00Z)
• Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese [55.95225353842118]
  We construct a large-scale dataset of image-text pairs in Chinese, where most data are retrieved from publicly available datasets. We develop 5 Chinese CLIP models of multiple sizes, spanning from 77 to 958 million parameters. Our experiments demonstrate that Chinese CLIP can achieve the state-of-the-art performance on MUGE, Flickr30K-CN, and COCO-CN.
  arXiv  Detail & Related papers  (2022-11-02T17:47:23Z)
• CPM: A Large-scale Generative Chinese Pre-trained Language Model [76.65305358932393]
  We release the Chinese Pre-trained Language Model (CPM) with generative pre-training on large-scale Chinese training data. CPM achieves strong performance on many NLP tasks in the settings of few-shot (even zero-shot) learning.
  arXiv  Detail & Related papers  (2020-12-01T11:32:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.