AraSSM: A bidirectional state-space encoder for Arabic masked language modeling

• URL: http://arxiv.org/abs/2608.08256v1
• Date: Sat, 08 Aug 2026 17:33:41 GMT
• Title: AraSSM: A bidirectional state-space encoder for Arabic masked language modeling
• Abstract summary: AraSSM is a bidirectional Mamba encoder pretrained via masked language modeling on a corpus combining Arabic Wikipedia and CulturaX text.<n>We introduce AraSSM, a bidirectional Mamba encoder pretrained via masked language modeling on a corpus combining Arabic Wikipedia and CulturaX text.
• Score: 0.19921702137386235
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretrained Transformer encoders such as AraBERT, MARBERT, and CAMeLBERT have become the standard backbone for Arabic natural language understanding, but their self-attention mechanism scales quadratically with sequence length, which limits efficiency on long documents. Mamba, a selective state-space model (SSM), offers linear-time sequence modeling as a competitive alternative to attention, yet no dedicated bidirectional Mamba encoder pretrained specifically for Arabic currently exists. We introduce AraSSM, a bidirectional Mamba encoder pretrained via masked language modeling on a corpus combining Arabic Wikipedia and CulturaX text, trained end-to-end on four consumer-grade NVIDIA RTX 2080Ti GPUs (11GB) over approximately ten days. We evaluate AraSSM by fine-tuning on four established Arabic NLU benchmarks covering sentiment classification (HARD), named entity recognition (ANERcorp), extractive question answering (ARCD), and natural language inference (XNLI-ar), following the per-task evaluation protocol introduced by AraBERT, and report results as mean +/- standard deviation across three fine-tuning seeds. AraSSM matches or exceeds published base-sized Transformer baselines on sentiment classification (96.37 +/- 0.03% accuracy on HARD), is competitive on extractive QA (32.19 +/- 1.07 EM, 63.79 +/- 0.25 F1 on ARCD) and named entity recognition (81.54 +/- 0.30 entity-level F1 on ANERcorp), and trails the base-sized Transformer range on natural language inference (72.83 +/- 0.07% accuracy on XNLI-ar), despite being trained entirely from scratch on consumer hardware rather than large-scale accelerator clusters.

Related papers

• A Sovereign, Open-Source Foundation Model for German and English [50.08096276509294]
  Soofi S 30B-A3B is a sovereign, open-source Mamba Transformer foundation model for German and English.<n>Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows.
  arXiv  Detail & Related papers  (2026-07-10T13:51:41Z)
• StereoFactory: A Unified Merging Framework for Robust Stereo Matching [61.973843344605655]
  Stereo matching has advanced through foundation models trained on large-scale datasets, yet this paradigm suffers from a scalability bottleneck.<n>Model merging offers a scalable post-hoc alternative by integrating knowledge from specialized models after source checkpoints are available.<n>We propose StereoFactory, a coarse-to-fine evolutionary framework for adaptive model merging.
  arXiv  Detail & Related papers  (2026-06-16T03:36:59Z)
• MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection [0.7039739305668745]
  This study proposes a two-phase framework for Arabic mental health text classification.<n>In phase 1, three Arabic pre-trained language models undergo Domain-Adaptive and Task-Adaptive Pretraining.<n>In phase 2, the selected model is assessed across four configurations with full fine-tuning and Low-Rank Adaptation.
  arXiv  Detail & Related papers  (2026-06-10T20:20:41Z)
• RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment [0.0]
  We present RightNow-Arabic-0.5B-Turbo, a 518M- parameter Arabic-specialized decoder built on Qwen2.5-0.5B.<n>The pipeline adds 27,032 Arabic tokens via mean-subtoken.<n>The merged model reaches 35.9% mean accuracy, beats every same-class open model, and recovers 67% of SILMA-9B's mean at 1/18 parameters.
  arXiv  Detail & Related papers  (2026-04-10T00:56:50Z)
• Hybrid CNN-Transformer Architecture for Arabic Speech Emotion Recognition [0.0]
  We present an Arabic Speech Emotion Recognition (SER) system based on a hybrid CNN-Transformer architecture.<n>The proposed model achieved 97.8% accuracy and a macro F1-score of 0.98.
  arXiv  Detail & Related papers  (2026-03-28T16:11:56Z)
• GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification [0.0]
  This paper presents system description for Arabic medical text classification across 82 distinct categories.<n>We benchmark this approach against a suite of multilingual and Arabic-specific encoders, as well as several large-scale causal decoders.<n> specialized bidirectional encoders significantly outperform causal decoders in capturing the precise semantic boundaries.
  arXiv  Detail & Related papers  (2026-02-17T16:13:08Z)
• Qwen3-ASR Technical Report [71.87071808763484]
  We introduce Qwen3-ASR family, which includes two powerful all-in-one speech recognition models and a novel non-autoregressive speech forced alignment model.<n>Qwen3-ASR-1.7B and Qwen3-ASR-0.6B are ASR models that support language identification and ASR for 52 languages and dialects.
  arXiv  Detail & Related papers  (2026-01-29T06:58:13Z)
• TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish [0.7233065479782755]
  TabiBERT is a monolingual Turkish encoder based on ModernBERT architecture trained from scratch on a large, curated corpus.<n>It supports 8,192-token context length (16x original BERT), achieves up to 2.65x speedup, and reduces GPU memory consumption.<n>It attains 77.58 on TabiBench, outperforming BERTurk by 1.62 points and establishing state-of-the-art on five of eight categories.
  arXiv  Detail & Related papers  (2025-12-28T20:18:22Z)
• mmBERT: A Modern Multilingual Encoder with Annealed Language Learning [57.58071656545661]
  mmBERT is an encoder-only language model pretrained on 3T tokens of multilingual text.<n>We add over 1700 low-resource languages to the data mix only during the decay phase.<n>We show that mmBERT significantly outperforms the previous generation of models on classification and retrieval tasks.
  arXiv  Detail & Related papers  (2025-09-08T17:08:42Z)
• LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA [4.104443734934105]
  We compare encoder-based Transformers (Chinese BERT-large and RoBERTa-wwm-ext-large), a decoder-only LLM (Alibaba's Qwen2.5-7B/Deep-R1-Distill-Qwen-7B fine-tuned via Low-Rank Adaptation, LoRA), and a FastText baseline.<n>Experiments reveal that although encoder models nearly memorize training data, they suffer significant performance degradation under distribution shifts.
  arXiv  Detail & Related papers  (2025-08-31T07:51:22Z)
• The Mamba in the Llama: Distilling and Accelerating Hybrid Models [76.64055251296548]
  We show how to distill large Transformers into linear RNNs by reusing the linear projection weights from attention layers with academic GPU resources.<n>The resulting hybrid model achieves performance comparable to the original Transformer in chat benchmarks.<n>We also introduce a hardware-aware speculative decoding algorithm that accelerates the inference speed of Mamba and hybrid models.
  arXiv  Detail & Related papers  (2024-08-27T17:56:11Z)
• Explicit Alignment Objectives for Multilingual Bidirectional Encoders [111.65322283420805]
  We present a new method for learning multilingual encoders, AMBER (Aligned Multilingual Bi-directional EncodeR) AMBER is trained on additional parallel data using two explicit alignment objectives that align the multilingual representations at different granularities. Experimental results show that AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLMR-large model.
  arXiv  Detail & Related papers  (2020-10-15T18:34:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.