Domain-specific Pretraining Profile and Transformer Performance: Evidence from Modeling Digital Pragmatics in Arabic-English Code-switching

• URL: http://arxiv.org/abs/2609.14571v1
• Date: Sun, 13 Sep 2026 14:58:38 GMT
• Title: Domain-specific Pretraining Profile and Transformer Performance: Evidence from Modeling Digital Pragmatics in Arabic-English Code-switching
• Abstract summary: This study highlights the role of domain-specific pretraining profile (DSPP) in Transformer performance for modeling digital pragmatics in Arabic-English code-switched discourse.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: This study highlights the role of domain-specific pretraining profile (DSPP) in Transformer performance for modeling digital pragmatics in Arabic-English code-switched discourse. It evaluates MARBERT and XLM-R(oBERTa), with BERT serving as a general-purpose baseline. The models were evaluated on their ability to classify context-sensitive pragmatic functions in code-switched social-media discourse. 11695 unique X posts were collected via Python and utilized for the study. The study employs a quantitative and qualitative NLP approach, following a supervised pipeline. Findings unveil that MARBERT consistently surpasses XLM-R with validation Macro F1 increasing from 0.39 to 0.84 and validation loss decreasing from 0.55 to 0.19. On an independent test set, it achieved 0.96 accuracy, 0.83 macro precision, 0.87 macro recall, and 0.85 Macro F1, while XLM-R achieved 0.92 test accuracy but a substantially lower Macro F1 of 0.52. This was also supported by class-level performance where MARBERT outperforms XLM-R considerably with F1 improvements ranging from +0.33 to +0.60, demonstrating a clear advantage in modeling Arabic digital pragmatics. The study concludes that Transformer performance depends more on DSPP than multilingual coverage alone, as the latter does not guarantee optimal performance on a highly specialized pragmatic classification task.

Related papers

• BERT-based Models vs. Large Language Models for Low-Resource Named Entity Recognition: A Comparative Study on Marathi [2.584263027095689]
  We fine-tune MahaBERT-v2 on different variants of the MahaNER dataset.<n>All models are evaluated on a Marathi NER test dataset using standard metrics of precision, recall, and F1-score.
  arXiv  Detail & Related papers  (2026-07-25T19:40:58Z)
• Enhancing Continual Learning for Software Vulnerability Prediction: Addressing Catastrophic Forgetting via Hybrid-Confidence-Aware Selective Replay for Temporal LLM Fine-Tuning [43.24339861841546]
  This paper investigates fine-tuning of a decoder-style language model (phi/phi-2 with LoRA) on a CVE-linked dataset.<n>We evaluate eight continual learning strategies, including window-only and cumulative training, replay-based baselines and regularisation-based variants.<n>Hybrid-CASR reduces training time per window by about 17 percent compared to the baseline, whereas cumulative training delivers only a minor F1 increase (0.661) at a 15.9-fold computational cost.
  arXiv  Detail & Related papers  (2026-02-27T09:13:23Z)
• Qwen3-ASR Technical Report [71.87071808763484]
  We introduce Qwen3-ASR family, which includes two powerful all-in-one speech recognition models and a novel non-autoregressive speech forced alignment model.<n>Qwen3-ASR-1.7B and Qwen3-ASR-0.6B are ASR models that support language identification and ASR for 52 languages and dialects.
  arXiv  Detail & Related papers  (2026-01-29T06:58:13Z)
• OR-R1: Automating Modeling and Solving of Operations Research Optimization Problem via Test-Time Reinforcement Learning [44.346973471913856]
  We present OR-R1, a data-efficient training framework for automated optimization modeling and solving.<n>Experiments show that OR-R1 achieves state-of-the-art performance with an average solving accuracy of $67.7%$.
  arXiv  Detail & Related papers  (2025-11-12T08:05:31Z)
• Hope Speech Detection in Social Media English Corpora: Performance of Traditional and Transformer Models [2.137286893609844]
  This proposal evaluates traditional machine learning models and fine-tuned transformers for a hope speech dataset as train, development and test set.<n> Transformer models delivered better results, the best model achieved weighted precision of 0.82, weighted recall of 0.80, weighted F1 of 0.79, macro F1 of 0.79, and 0.80 accuracy.
  arXiv  Detail & Related papers  (2025-10-27T17:53:40Z)
• ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs [75.72672339168092]
  We introduce ReasonFlux-PRM, a novel trajectory-aware PRM to evaluate trajectory-response type of reasoning traces.<n>ReasonFlux-PRM incorporates both step-level and trajectory-level supervision, enabling fine-grained reward assignment aligned with structured chain-of-thought data.<n>Our derived ReasonFlux-PRM-7B yields consistent performance improvements, achieving average gains of 12.1% in supervised fine-tuning, 4.5% in reinforcement learning, and 6.3% in test-time scaling.
  arXiv  Detail & Related papers  (2025-06-23T17:59:02Z)
• R-PRM: Reasoning-Driven Process Reward Modeling [53.06844294668382]
  Process Reward Models (PRMs) have emerged as a promising solution by evaluating each reasoning step.<n>Existing PRMs typically output evaluation scores directly, limiting both learning efficiency and evaluation accuracy.<n>We propose Reasoning-Driven Process Reward Modeling (R-PRM)<n>R-PRM generates seed data from limited annotations, effectively bootstrapping our model's reasoning capabilities.
  arXiv  Detail & Related papers  (2025-03-27T09:23:08Z)
• SPARC: Score Prompting and Adaptive Fusion for Zero-Shot Multi-Label Recognition in Vision-Language Models [74.40683913645731]
  Zero-shot multi-label recognition (MLR) with Vision-Language Models (VLMs) faces significant challenges without training data, model tuning, or architectural modifications.<n>Our work proposes a novel solution treating VLMs as black boxes, leveraging scores without training data or ground truth.<n>Analysis of these prompt scores reveals VLM biases and AND''/OR' signal ambiguities, notably that maximum scores are surprisingly suboptimal compared to second-highest scores.
  arXiv  Detail & Related papers  (2025-02-24T07:15:05Z)
• Comparing Hyper-optimized Machine Learning Models for Predicting Efficiency Degradation in Organic Solar Cells [38.647921189039934]
  This work presents a set of optimal machine learning (ML) models to represent the temporal degradation suffered by the power conversion efficiency (PCE) of organic solar cells (OSCs)<n>We generated a database with 996 entries, which includes up to 7 variables regarding both the manufacturing process and environmental conditions for more than 180 days.<n>The accuracy achieved reaches values of the coefficient determination (R2) widely exceeding 0.90, whereas the root mean squared error (RMSE), sum of squared error (SSE), and mean absolute error (MAE)>1% of the target value, the PCE.
  arXiv  Detail & Related papers  (2024-03-29T22:05:26Z)
• MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers [41.56951365163419]
  "MixedNUTS" is a training-free method where the output logits of a robust classifier are processed by nonlinear transformations with only three parameters. MixedNUTS then converts the transformed logits into probabilities and mixes them as the overall output. On CIFAR-10, CIFAR-100, and ImageNet datasets, experimental results with custom strong adaptive attacks demonstrate MixedNUTS's vastly improved accuracy and near-SOTA robustness.
  arXiv  Detail & Related papers  (2024-02-03T21:12:36Z)
• Low-resource classification of mobility functioning information in clinical sentences using large language models [0.0]
  This study evaluates the ability of publicly available large language models (LLMs) to accurately identify the presence of functioning information from clinical notes. We collect a balanced binary classification dataset of 1000 sentences from the Mobility NER dataset, which was curated from n2c2 clinical notes.
  arXiv  Detail & Related papers  (2023-12-15T20:59:17Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)
• Exploring the Value of Pre-trained Language Models for Clinical Named Entity Recognition [6.917786124918387]
  We compare Transformer models that are trained from scratch to fine-tuned BERT-based LLMs. We examine the impact of an additional CRF layer on such models to encourage contextual learning.
  arXiv  Detail & Related papers  (2022-10-23T16:27:31Z)
• HinglishNLP: Fine-tuned Language Models for Hinglish Sentiment Detection [0.0]
  This work adds two common approaches to sentiment analysis. NB-SVM beats RoBERTa by 6.2% (relative) F1. The best performing model is a majority-vote ensemble which achieves an F1 of 0.707.
  arXiv  Detail & Related papers  (2020-08-22T12:01:44Z)
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention [119.77305080520718]
  We propose a new model architecture DeBERTa that improves the BERT and RoBERTa models using two novel techniques. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understanding (NLU) and natural langauge generation (NLG) downstream tasks.
  arXiv  Detail & Related papers  (2020-06-05T19:54:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.