MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection

• URL: http://arxiv.org/abs/2606.12649v1
• Date: Wed, 10 Jun 2026 20:20:41 GMT
• Title: MentalMARBERT: Domain-Adaptive Pre-training and Two-Stage Fine-Tuning for Arabic Mental Health Disorders Detection
• Abstract summary: This study proposes a two-phase framework for Arabic mental health text classification.<n>In phase 1, three Arabic pre-trained language models undergo Domain-Adaptive and Task-Adaptive Pretraining.<n>In phase 2, the selected model is assessed across four configurations with full fine-tuning and Low-Rank Adaptation.
• Score: 0.7039739305668745
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Detecting mental health disorders from Arabic social media text remains challenging due to dialectal variation, informal language, limited high-quality annotated resources, and severe class imbalance. While English mental health natural language processing (NLP) has progressed substantially, Arabic multi-class disorder classification remains insufficiently studied. This study proposes a two-phase framework for Arabic mental health text classification. In phase 1, three Arabic pre-trained language models, AraBERT, CAMeLBERT, and MARBERT, undergo Domain-Adaptive and Task-Adaptive Pretraining (DAPT and TAPT) using a large-scale corpus of unlabeled Arabic mental health tweets. The adapted models are evaluated under a unified protocol to identify the most effective backbone model. In phase 2, the selected model is assessed across four configurations combining single-stage and hierarchical two-stage classification architectures with full fine-tuning and Low-Rank Adaptation (LoRA). To support this study, we constructed a novel annotated Arabic mental health dataset comprising 50,670 tweets across six categories, with strong inter annotator agreement (Krippendorff's Alpha = 0.733, average pairwise agreement = 0.797). Experimental results show that the domain-adapted MARBERT (MentalMARBERT) achieves statistically significant improvements over baseline models in both accuracy and macro-F1. The hierarchical two-stage architecture combined with full fine-tuning achieves the best overall performance, reaching a macro-F1 of 0.861 and an accuracy of 0.877. These findings demonstrate the effectiveness of domain-specific adaptive pretraining and hierarchical classification for Arabic mental health disorder detection.

Related papers

• ArabicDialectSafety: A Dialect-Aware Benchmark for Arabic Content Safety Classification [1.4665143197624044]
  We present a human-curated Arabic safety dataset of 25,071 prompts covering six Arabic varieties.<n>The dataset is annotated with dialect labels and seven fine-grained harm categories.<n> Benchmarking seven supervised and generative models, we find that fine-tuned MARBERTv2 achieves the strongest performance.
  arXiv  Detail & Related papers  (2026-08-02T15:00:58Z)
• Automated ICD Classification of Psychiatric Diagnoses: From Classical NLP to Large Language Models [67.51123543731806]
  Mental health has become a global priority, leading to a massive administrative burden in the coding of clinical diagnoses.<n>This study proposes the automation of psychiatric diagnostic analysis by mapping free-text descriptions to the International Classification of Diseases (ICD) using Natural Language Processing (NLP) and Machine Learning (ML) techniques.
  arXiv  Detail & Related papers  (2026-05-20T13:26:05Z)
• Severity-Aware Weighted Loss for Arabic Medical Text Generation [0.25489046505746704]
  We propose a severity-aware weighted loss for fine-tuning Arabic language models on medical complaint-response data.<n>Experiments are conducted using the MAQA dataset, which provides Arabic medical complaints and trusted human responses.
  arXiv  Detail & Related papers  (2026-04-07T18:26:41Z)
• Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages [0.0]
  This study evaluates a two-step pipeline combining Aya-expanse-8B as a Persian-to-English translation model with five open-source small language models (SLMs)<n>Models were assessed on macro-averaged F1-score, Matthews Correlation Coefficient (MCC), sensitivity, and specificity to account for class imbalance.<n>A bilingual analysis of Aya-expanse-8B revealed that translating Persian transcripts to English improved sensitivity, reduced missing outputs, and boosted metrics robust to class imbalance.
  arXiv  Detail & Related papers  (2026-02-24T21:10:29Z)
• Qwen3-ASR Technical Report [71.87071808763484]
  We introduce Qwen3-ASR family, which includes two powerful all-in-one speech recognition models and a novel non-autoregressive speech forced alignment model.<n>Qwen3-ASR-1.7B and Qwen3-ASR-0.6B are ASR models that support language identification and ASR for 52 languages and dialects.
  arXiv  Detail & Related papers  (2026-01-29T06:58:13Z)
• Binary Token-Level Classification with DeBERTa for All-Type MWE Identification: A Lightweight Approach with Linguistic Enhancement [1.8429656136522097]
  We present a comprehensive approach for multiword expression (MWE) identification that combines binary token-level classification, linguistic feature integration, and data augmentation.<n>Our DeBERTa-v3-large model achieves 69.8% F1 on the CoAM dataset, surpassing the best results (Qwen-72B, 57.8% F1) on this dataset by 12 points while using 165x fewer parameters.
  arXiv  Detail & Related papers  (2026-01-27T08:42:54Z)
• Hierarchical Self-Supervised Representation Learning for Depression Detection from Speech [51.14752758616364]
  Speech-based depression detection (SDD) is a promising, non-invasive alternative to traditional clinical assessments.<n>We propose HAREN-CTC, a novel architecture that integrates multi-layer SSL features using cross-attention within a multitask learning framework.<n>The model achieves state-of-the-art macro F1-scores of 0.81 on DAIC-WOZ and 0.82 on MODMA, outperforming prior methods across both evaluation scenarios.
  arXiv  Detail & Related papers  (2025-10-05T09:32:12Z)
• Mental Multi-class Classification on Social Media: Benchmarking Transformer Architectures against LSTM Models [7.464241214592479]
  We present a large-scale comparative study of state-of-the-art transformer versus Long Short-Term Memory (LSTM)-based models to classify mental health posts.<n>We first curate a large dataset of Reddit posts spanning six mental health conditions and a control group, using rigorous filtering and statistical exploratory analysis to ensure annotation quality.<n> Experimental results show that transformer models consistently outperform the alternatives, with RoBERTa achieving 91-99% F1-scores and accuracies across all classes.
  arXiv  Detail & Related papers  (2025-09-20T05:41:59Z)
• Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval [49.1574468325115]
  We introduce Amharic-specific dense retrieval models based on pre-trained Amharic BERT and RoBERTa backbones.<n>Our proposed RoBERTa-Base-Amharic-Embed model (110M parameters) achieves a 17.6% relative improvement in MRR@10.<n>More compact variants, such as RoBERTa-Medium-Amharic-Embed (42M) remain competitive while being over 13x smaller.
  arXiv  Detail & Related papers  (2025-05-25T23:06:20Z)
• A Comprehensive Evaluation of Large Language Models on Mental Illnesses in Arabic Context [0.9074663948713616]
  Mental health disorders pose a growing public health concern in the Arab world.<n>This study comprehensively evaluates 8 large language models (LLMs) on diverse mental health datasets.
  arXiv  Detail & Related papers  (2025-01-12T16:17:25Z)
• SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment [78.4550589538805]
  We propose an efficient multilingual reasoning alignment approach that precisely identifies and fine-tunes the layers responsible for handling multilingualism.<n> Experimental results show that our method, SLAM, only tunes 6 layers' feed-forward sub-layers including 6.5-8% of all parameters within 7B and 13B LLMs.
  arXiv  Detail & Related papers  (2025-01-07T10:29:43Z)
• Automatically measuring speech fluency in people with aphasia: first achievements using read-speech data [55.84746218227712]
  This study aims at assessing the relevance of a signalprocessingalgorithm, initially developed in the field of language acquisition, for the automatic measurement of speech fluency.
  arXiv  Detail & Related papers  (2023-08-09T07:51:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.