Parameter-Efficient Fine-Tuning for Low-Resource Languages: A Comparative Study of LLMs for Bengali Hate Speech Detection

• URL: http://arxiv.org/abs/2510.16985v1
• Date: Sun, 19 Oct 2025 20:03:22 GMT
• Title: Parameter-Efficient Fine-Tuning for Low-Resource Languages: A Comparative Study of LLMs for Bengali Hate Speech Detection
• Abstract summary: This paper presents the first application of Efficient Fine-Tuning (PEFT) for Bengali hate speech detection using LoRA and QLoRA.<n>Three instruction-tuned large language models were fine-tuned on the BD-SHS dataset of 50,281 annotated comments.<n>Llama-3.2-3B achieved the highest F1-score of 92.23%, followed by Mistral-7B at 88.94% and Gemma-3-4B at 80.25%.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bengali social media platforms have witnessed a sharp increase in hate speech, disproportionately affecting women and adolescents. While datasets such as BD-SHS provide a basis for structured evaluation, most prior approaches rely on either computationally costly full-model fine-tuning or proprietary APIs. This paper presents the first application of Parameter-Efficient Fine-Tuning (PEFT) for Bengali hate speech detection using LoRA and QLoRA. Three instruction-tuned large language models - Gemma-3-4B, Llama-3.2-3B, and Mistral-7B - were fine-tuned on the BD-SHS dataset of 50,281 annotated comments. Each model was adapted by training fewer than 1% of its parameters, enabling experiments on a single consumer-grade GPU. The results show that Llama-3.2-3B achieved the highest F1-score of 92.23%, followed by Mistral-7B at 88.94% and Gemma-3-4B at 80.25%. These findings establish PEFT as a practical and replicable strategy for Bengali and related low-resource languages.

Related papers

• GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models [0.06157382820537719]
  The evaluation of mathematical reasoning in large language models (LLMs) has predominantly focused on high-resource languages like English.<n>This study introduces GSM-Plus-BN, a novel perturbated Bengali mathematical dataset.<n>We evaluate six open-source LLMs Qwen3-32B, Llama-3.1-8B-Instant, Llama-3.3-70B-Versatile, Llama-4-Scout-17B-16E-Instruct, GPT-OSS-120B, and GPT-OSS-20B using a benchmark of 9,000 evaluation samples.
  arXiv  Detail & Related papers  (2026-07-14T20:23:27Z)
• Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir [0.0]
  This paper presents a comparative study of parameter-efficient fine-tuning methods, including LoRA and QLoRA, applied to the task of adapting large language models to the Bashkir language.<n> Experimental evaluation is conducted on a Bashkir text corpus of 71k documents (46.9M tokens) using models of various architectures.
  arXiv  Detail & Related papers  (2026-05-06T14:14:35Z)
• Benchmarking Linguistic Adaptation in Comparable-Sized LLMs: A Study of Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B on Romanized Nepali [0.0]
  Romanized Nepali, the Nepali language written in the Latin alphabet, is the dominant medium for informal digital communication in Nepal.<n>This study presents a systematic benchmarking of linguistic adaptation across three comparable-sized open-weight models.
  arXiv  Detail & Related papers  (2026-03-25T07:02:51Z)
• Bengali Text Classification: An Evaluation of Large Language Model Approaches [0.0]
  Unlike English, Bengali faces challenges due to the lack of extensive annotated datasets and pre-trained language models.<n>This study explores the effectiveness of large language models (LLMs) in classifying Bengali newspaper articles.
  arXiv  Detail & Related papers  (2026-01-17T18:25:19Z)
• Gradient Masters at BLP-2025 Task 1: Advancing Low-Resource NLP for Bengali using Ensemble-Based Adversarial Training for Hate Speech Detection [1.2744523252873352]
  We present an ensemble-based fine-tuning strategy for addressing subtasks 1A (hate-type classification) and 1B (target group classification) in YouTube comments.<n>We propose a hybrid approach on a Bangla Language Model, which outperformed the baseline models and secured the 6th position in subtask 1A.
  arXiv  Detail & Related papers  (2025-11-23T07:29:09Z)
• LLM Based Sentiment Classification From Bangladesh E-Commerce Reviews [0.0]
  The viability of using transformer-based BERT models for sentiment analysis from Bangladesh e commerce reviews is investigated in this paper.<n>A subset of 4000 samples from the original dataset of Bangla and English customer reviews was utilized to fine-tune the model.<n>The fine tuned Llama-3.1-8B model outperformed other fine-tuned models, with an overall accuracy, precision, recall, and F1 score of 95.5%, 93%, 88%, 90%.
  arXiv  Detail & Related papers  (2025-09-30T16:46:09Z)
• Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval [49.1574468325115]
  We introduce Amharic-specific dense retrieval models based on pre-trained Amharic BERT and RoBERTa backbones.<n>Our proposed RoBERTa-Base-Amharic-Embed model (110M parameters) achieves a 17.6% relative improvement in MRR@10.<n>More compact variants, such as RoBERTa-Medium-Amharic-Embed (42M) remain competitive while being over 13x smaller.
  arXiv  Detail & Related papers  (2025-05-25T23:06:20Z)
• MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining [60.02032710118597]
  We present MiMo-7B, a large language model born for reasoning tasks, with optimization across both pre-training and post-training stages.<n>MiMo-7B-Base is pre-trained on 25 trillion tokens, with additional Multi-Token Prediction objective for enhanced performance and accelerated inference speed.<n>The final RL-tuned model, MiMo-7B-RL, achieves superior performance on mathematics, code and general reasoning tasks, surpassing the performance of OpenAI o1-mini.
  arXiv  Detail & Related papers  (2025-05-12T14:30:11Z)
• Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarization [1.2289361708127877]
  This study investigates the zero-shot performance of nine advanced large language models (LLMs)<n>We benchmarked these LLMs using ROUGE metrics against Bangla T5, a fine-tuned state-of-the-art model.<n>The results demonstrate that zero-shot LLMs can rival fine-tuned models, achieving high-quality summaries even without task-specific training.
  arXiv  Detail & Related papers  (2025-05-08T09:06:28Z)
• Leveraging Prompt-Tuning for Bengali Grammatical Error Explanation Using Large Language Models [0.0]
  We propose a novel three-step prompt-tuning method for Bengali Grammatical Error Explanation (BGEE)<n>Our approach involves identifying and categorizing grammatical errors in Bengali sentences, generating corrected versions of the sentences, and providing natural language explanations for each identified error.<n>We evaluate the performance of our BGEE system using both automated evaluation metrics and human evaluation conducted by experienced Bengali language experts.
  arXiv  Detail & Related papers  (2025-04-08T03:38:01Z)
• SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment [78.4550589538805]
  We propose an efficient multilingual reasoning alignment approach that precisely identifies and fine-tunes the layers responsible for handling multilingualism.<n> Experimental results show that our method, SLAM, only tunes 6 layers' feed-forward sub-layers including 6.5-8% of all parameters within 7B and 13B LLMs.
  arXiv  Detail & Related papers  (2025-01-07T10:29:43Z)
• Benchmarking the Performance of Pre-trained LLMs across Urdu NLP Tasks [0.9786690381850356]
  This study presents in-depth examination of 7 prominent Large Language Models (LLMs) across 17 tasks using 22 datasets, 13.8 hours of speech, in a zero-shot setting, and their performance against state-of-the-art (SOTA) models.<n>Our results emphasize that models with fewer parameters but richer language-specific data, like Llama 3.1-8B, often outperform larger models with lower language diversity, such as GPT-3.5, in several tasks.
  arXiv  Detail & Related papers  (2024-05-24T11:30:37Z)
• Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations [59.056367787688146]
  This paper pioneers exploring and training powerful Multilingual Math Reasoning (xMR) LLMs. We construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages. By utilizing translation, we construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages.
  arXiv  Detail & Related papers  (2023-10-31T08:09:20Z)
• LAraBench: Benchmarking Arabic AI with Large Language Models [26.249084464525044]
  LAraBench addresses this gap for Arabic Natural Language Processing (NLP) and Speech Processing tasks. We utilize models such as GPT-3.5-turbo, GPT-4, BLOOMZ, Jais-13b-chat, Whisper, and USM to tackle 33 distinct tasks across 61 publicly available datasets. This involved 98 experimental setups, encompassing 296K data points, 46 hours of speech, and 30 sentences for Text-to-Speech (TTS)
  arXiv  Detail & Related papers  (2023-05-24T10:16:16Z)
• Text Classification via Large Language Models [63.1874290788797]
  We introduce Clue And Reasoning Prompting (CARP) to address complex linguistic phenomena involved in text classification. Remarkably, CARP yields new SOTA performances on 4 out of 5 widely-used text-classification benchmarks. More importantly, we find that CARP delivers impressive abilities on low-resource and domain-adaptation setups.
  arXiv  Detail & Related papers  (2023-05-15T06:24:45Z)
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts [84.33607245023049]
  We propose and develop a family of language models named GLaM (Generalist Language Model) GLaM uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the flops for inference, while still achieving better overall zero-shot and one-shot performance across 29 NLP tasks.
  arXiv  Detail & Related papers  (2021-12-13T18:58:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.