A Multi-Model Approach to English-Bangla Sentiment Classification of Government Mobile Banking App Reviews

• URL: http://arxiv.org/abs/2604.13057v1
• Date: Tue, 17 Mar 2026 21:43:51 GMT
• Title: A Multi-Model Approach to English-Bangla Sentiment Classification of Government Mobile Banking App Reviews
• Abstract summary: The study analyzed 5,652 Google Play reviews in English and Bangla for four Bangladeshi government banking apps.<n>Traditional models produced the highest accuracy (0.815), while Linear SVM produced the highest weighted F1 score (0.804)<n>DeBERTa-v3 was applied to analyze the sentiment at the aspect level across the reviews for the four apps.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: For millions of users in developing economies who depend on mobile banking as their primary gateway to financial services, app quality directly shapes financial access. The study analyzed 5,652 Google Play reviews in English and Bangla (filtered from 11,414 raw reviews) for four Bangladeshi government banking apps. The authors used a hybrid labeling approach that combined use of the reviewer's star rating for each review along with a separate independent XLM-RoBERTa classifier to produce moderate inter-method agreement (kappa = 0.459). Traditional models outperformed transformer-based ones: Random Forest produced the highest accuracy (0.815), while Linear SVM produced the highest weighted F1 score (0.804); both were higher than the performance of fine-tuned XLM-RoBERTa (0.793). McNemar's test confirmed that all classical models were significantly superior to the off-the-shelf XLM-RoBERTa (p < 0.05), while differences with the fine-tuned variant were not statistically significant. DeBERTa-v3 was applied to analyze the sentiment at the aspect level across the reviews for the four apps; the reviewers expressed their dissatisfaction primarily with the speed of transactions and with the poor design of interfaces; eJanata app received the worst ratings from the reviewers across all apps. Three policy recommendations are made based on these findings - remediation of app quality, trust-centred release management, and Bangla-first NLP adoption - to assist state-owned banks in moving towards improving their digital services through data-driven methods. Notably, a 16.1-percentage-point accuracy gap between Bangla and English text highlights the need for low-resource language model development.

Related papers

• Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge [46.547776251310694]
  This paper evaluates 5G domain understanding and fault analysis in a free-text generation format.<n>We evaluate three lightweight LLMs, Claude-Haiku-4.5, GPT-5.4-Mini, and Gemini-3.1-Flash-Lite, on free-text 5G domain knowledge and fault-analysis tasks.
  arXiv  Detail & Related papers  (2026-08-21T12:09:51Z)
• Can Open-Weight Models Compete on Financial Text Comprehension? [0.0]
  Open-weight language models from Chinese AI labs caught up on benchmarks relative to proprietary frontier models in recent months.<n>Yet their reliability on real-world financial tasks remains largely untested.<n>We update the Financial Touchstone benchmark, which now has 2,967 question context-answer triplets across 495 international annual reports.
  arXiv  Detail & Related papers  (2026-08-09T10:51:52Z)
• SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators [50.129606229339146]
  SurveyReview is a reviewer-aligned, multi-dimensional benchmark and dataset for survey evaluation.<n>We structure authentic peer-review reports by converting free-form comments into four-dimensional scores.<n>We release standardized train/test splits and an evaluation protocol to measure alignment between automatic evaluators and human reviewers.
  arXiv  Detail & Related papers  (2026-08-07T16:11:46Z)
• From Financial Sentiment Classification to Return Predictability: A QLoRA Benchmark of Large Language Models [0.0]
  We construct a unified three-class benchmark from five financial text datasets.<n>We compare TF--IDF Naive Bayes, off-the-shelf FinBERT and Financial-RoBERTa encoders.<n>We evaluate economic validity on a temporally separate 2019 Benzinga sample.
  arXiv  Detail & Related papers  (2026-08-04T19:57:40Z)
• InvestPhilBench: A Multi-Layer Dynamic Benchmark for Evaluating Large Language Model Procedural Reasoning in Expert Investment Philosophy [0.9924185669370708]
  InvestPhilBench is a dynamic benchmark spanning eight cognitive tiers.<n>The v0.6 release comprises 118 primary-source-verified investment principle cards.<n>For reproducible scoring at scale we introduce the Benchmark Automated Scoring Pipeline.
  arXiv  Detail & Related papers  (2026-06-24T15:53:20Z)
• Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks [45.86413490112477]
  Large language models (LLMs) are used in reliability-sensitive applications.<n>Estimateing rare failure probability with tight confidence bounds requires prohibitively large LLM inference sizes.<n>We propose to learn a sampling distribution concentrated on failure-prone inputs via the cross-entropy method.
  arXiv  Detail & Related papers  (2026-05-11T20:23:44Z)
• IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text [0.0]
  We introduce IndiaFinBench, the first publicly available evaluation benchmark for assessing large language model (LLM) performance on Indian financial regulatory text.<n>IndiaFinBench addresses this gap with 406 expert-annotated question-answer pairs drawn from 192 documents sourced from the Securities and Exchange Board of India and the Reserve Bank of India.<n>We evaluate twelve models under zero-shot conditions, with accuracy ranging from 70.4% (Gemma 4 E4B) to 89.7% (Gemini 2.5 Flash)
  arXiv  Detail & Related papers  (2026-04-21T10:04:49Z)
• Calibrated Confidence Estimation for Tabular Question Answering [0.0]
  All models are severely overconfident (smooth ECE 0.35-0.64 versus 0.10-0.15 reported for textual QA)<n>A consistent self-evaluation versus perturbation dichotomy replicates across both benchmarks and all four fully-covered models.<n>A secondary contribution, structure-aware recalibration, improves AUROC by percentage points over standard post-hoc methods.
  arXiv  Detail & Related papers  (2026-04-14T09:16:53Z)
• Qwen3-ASR Technical Report [71.87071808763484]
  We introduce Qwen3-ASR family, which includes two powerful all-in-one speech recognition models and a novel non-autoregressive speech forced alignment model.<n>Qwen3-ASR-1.7B and Qwen3-ASR-0.6B are ASR models that support language identification and ASR for 52 languages and dialects.
  arXiv  Detail & Related papers  (2026-01-29T06:58:13Z)
• Catching UX Flaws in Code: Leveraging LLMs to Identify Usability Flaws at the Development Stage [0.0]
  This paper investigates whether large language models (LLMs) can provide reliable and consistent assessments at the development stage.<n>We generated over 850 evaluations in three independent evaluations per site using a pipeline of OpenAI's GPT-4o.<n>For issue detection, the model demonstrated moderate consistency, with an average pairwise Cohen's Kappa of 0.50 and an exact agreement of 84%.
  arXiv  Detail & Related papers  (2025-12-03T21:02:54Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent [46.41047559759938]
  Computer-using agents (CUAs) enable task completion through natural interaction with operating systems and software interfaces.<n> Reward models offer promising alternatives, but their effectiveness on CUA evaluation remains largely underexplored.<n>We present CUARewardBench, comprising four key contributions.
  arXiv  Detail & Related papers  (2025-10-21T12:53:40Z)
• LLM Based Sentiment Classification From Bangladesh E-Commerce Reviews [0.0]
  The viability of using transformer-based BERT models for sentiment analysis from Bangladesh e commerce reviews is investigated in this paper.<n>A subset of 4000 samples from the original dataset of Bangla and English customer reviews was utilized to fine-tune the model.<n>The fine tuned Llama-3.1-8B model outperformed other fine-tuned models, with an overall accuracy, precision, recall, and F1 score of 95.5%, 93%, 88%, 90%.
  arXiv  Detail & Related papers  (2025-09-30T16:46:09Z)
• Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications [0.7124971549479361]
  This study introduces a framework for evaluating consistency in large language model (LLM) binary text classification.<n>We determine sample size requirements, develop metrics for invalid responses, and evaluate intra- and inter-rater reliability.
  arXiv  Detail & Related papers  (2025-05-20T21:12:58Z)
• Do Large Language Model Benchmarks Test Reliability? [66.1783478365998]
  We investigate how well current benchmarks quantify model reliability.<n>Motivated by this gap in the evaluation of reliability, we propose the concept of so-called platinum benchmarks.<n>We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks.
  arXiv  Detail & Related papers  (2025-02-05T18:58:19Z)
• ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models [12.035509884945789]
  We introduce a tuning-free framework called ReFeR, designed to evaluate generative outputs, including both text and images. We rigorously evaluate our framework, ReFeR, across four diverse evaluation tasks. Experiments on four reasoning tasks demonstrate superior collective reasoning abilities of the framework.
  arXiv  Detail & Related papers  (2024-07-16T08:25:26Z)
• Advancing LLM Reasoning Generalists with Preference Trees [119.57169648859707]
  We introduce Eurus, a suite of large language models (LLMs) optimized for reasoning. Eurus models achieve state-of-the-art results among open-source models on a diverse set of benchmarks.
  arXiv  Detail & Related papers  (2024-04-02T16:25:30Z)
• SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness [66.37077266814822]
  In natural language, it is difficult to generate new examples that stay on the underlying data manifold. We introduce SSMBA, a data augmentation method for generating synthetic training examples. In experiments on benchmarks across 3 tasks and 9 datasets, SSMBA consistently outperforms existing data augmentation methods.
  arXiv  Detail & Related papers  (2020-09-21T22:02:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.