Unlocking the Potential of Multiple BERT Models for Bangla Question Answering in NCTB Textbooks

• URL: http://arxiv.org/abs/2412.18440v1
• Date: Tue, 24 Dec 2024 13:59:23 GMT
• Title: Unlocking the Potential of Multiple BERT Models for Bangla Question Answering in NCTB Textbooks
• Abstract summary: evaluating text comprehension in educational settings is critical for understanding student performance and improving curricular effectiveness.<n>This study investigates the capability of state-of-the-art language models-RoBERTa Base, Bangla-BERT, and BERT Base-in automatically assessing Bangla passage-based question-answering from the National Curriculum and Textbook Board (NCTB) textbooks for classes 6-10.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Evaluating text comprehension in educational settings is critical for understanding student performance and improving curricular effectiveness. This study investigates the capability of state-of-the-art language models-RoBERTa Base, Bangla-BERT, and BERT Base-in automatically assessing Bangla passage-based question-answering from the National Curriculum and Textbook Board (NCTB) textbooks for classes 6-10. A dataset of approximately 3,000 Bangla passage-based question-answering instances was compiled, and the models were evaluated using F1 Score and Exact Match (EM) metrics across various hyperparameter configurations. Our findings revealed that Bangla-BERT consistently outperformed the other models, achieving the highest F1 (0.75) and EM (0.53) scores, particularly with smaller batch sizes, the inclusion of stop words, and a moderate learning rate. In contrast, RoBERTa Base demonstrated the weakest performance, with the lowest F1 (0.19) and EM (0.27) scores under certain configurations. The results underscore the importance of fine-tuning hyperparameters for optimizing model performance and highlight the potential of machine learning models in evaluating text comprehension in educational contexts. However, limitations such as dataset size, spelling inconsistencies, and computational constraints emphasize the need for further research to enhance the robustness and applicability of these models. This study lays the groundwork for the future development of automated evaluation systems in educational institutions, providing critical insights into model performance in the context of Bangla text comprehension.

Related papers

• CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming [0.48342038441006796]
  Large language models are increasingly explored as AI tutors, yet deploying them in K-12 settings raises concerns around privacy, cost, and reliance on proprietary models.<n>Small language models (SLMs) offer a promising alternative, but selecting the right model for a specific educational context remains difficult.<n>We introduce CSTutorBench, a benchmark for evaluating language models as CS tutors in VEX VR, a block-based robotics environment.
  arXiv  Detail & Related papers  (2026-07-06T19:15:07Z)
• Automatic Essay Scoring and Feedback Generation in Basque Language Learning [4.218073067465283]
  This paper introduces the first publicly available dataset for Automatic Essay Scoring (AES) and feedback generation in Basque, targeting the CEFR C1 proficiency level.<n>The dataset comprises 3,200 essays from HABE, each annotated by expert evaluators with criterion specific scores covering correctness, richness, coherence, cohesion, and task alignment enriched with detailed feedback and error examples.<n>We fine-tune open-source models, including RoBERTa-EusCrawl and Latxa 8B/70B, for both scoring and explanation generation.
  arXiv  Detail & Related papers  (2025-12-09T15:28:35Z)
• Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Review [18.045716459188366]
  Item difficulty plays a crucial role in test performance, interpretability of scores, and equity for all test-takers, especially in large-scale assessments.<n>Traditional approaches to item difficulty modeling rely on field testing and classical test theory (CTT)-based item analysis or item response theory (IRT) calibration.<n>This paper reviews and synthesizes 37 articles on automated item difficulty prediction in large-scale assessment settings published through May 2025.
  arXiv  Detail & Related papers  (2025-09-27T20:19:39Z)
• EduBench: A Comprehensive Benchmarking Dataset for Evaluating Large Language Models in Diverse Educational Scenarios [41.370448581863194]
  We introduce the first diverse benchmark tailored for educational scenarios.<n>We propose a set of multi-dimensional evaluation metrics that cover 12 critical aspects relevant to both teachers and students.<n>We train a relatively small-scale model on our constructed dataset and demonstrate it can achieve performance comparable to state-of-the-art large models.
  arXiv  Detail & Related papers  (2025-05-22T03:01:28Z)
• Unraveling the Capabilities of Language Models in News Summarization [0.0]
  This work provides a comprehensive benchmarking of 20 recent language models, focusing on smaller ones for the news summarization task. We focus in this study on zero-shot and few-shot learning settings and we apply a robust evaluation methodology. We highlight the exceptional performance of GPT-3.5-Turbo and GPT-4, which generally dominate due to their advanced capabilities.
  arXiv  Detail & Related papers  (2025-01-30T04:20:16Z)
• What Differentiates Educational Literature? A Multimodal Fusion Approach of Transformers and Computational Linguistics [0.7342677574855649]
  The integration of new literature into the English curriculum remains a challenge since educators often lack scalable tools to rapidly evaluate readability and adapt texts for diverse classroom needs.<n>This study proposes to address this gap through a multimodal approach that combines transformer-based text classification with linguistic feature analysis to align texts with UK Key Stages.<n>The proposed approach is finally encapsulated in a stakeholder-facing web application, providing non-technical stakeholder access to real-time insights on text complexity, reading difficulty, curriculum alignment, and recommendations for learning age range.
  arXiv  Detail & Related papers  (2024-11-26T17:01:27Z)
• The Surprising Effectiveness of Test-Time Training for Few-Shot Learning [59.309477460893916]
  Language models (LMs) have shown impressive performance on tasks within their training distribution, but often struggle with structurally novel tasks.<n>We investigate the effectiveness of test-time training (TTT) as a mechanism for improving LMs' reasoning and few-shot learning capabilities.<n>Our findings highlight the limitations of in-context learning for novel tasks and demonstrate the potential of test-time training to enhance language model adaptability.
  arXiv  Detail & Related papers  (2024-11-11T18:59:45Z)
• How Hard is this Test Set? NLI Characterization by Exploiting Training Dynamics [49.9329723199239]
  We propose a method for the automated creation of a challenging test set without relying on the manual construction of artificial and unrealistic examples. We categorize the test set of popular NLI datasets into three difficulty levels by leveraging methods that exploit training dynamics. When our characterization method is applied to the training set, models trained with only a fraction of the data achieve comparable performance to those trained on the full dataset.
  arXiv  Detail & Related papers  (2024-10-04T13:39:21Z)
• CELA: Cost-Efficient Language Model Alignment for CTR Prediction [70.65910069412944]
  Click-Through Rate (CTR) prediction holds a paramount position in recommender systems.<n>Recent efforts have sought to mitigate these challenges by integrating Pre-trained Language Models (PLMs)<n>We propose textbfCost-textbfEfficient textbfLanguage Model textbfAlignment (textbfCELA) for CTR prediction.
  arXiv  Detail & Related papers  (2024-05-17T07:43:25Z)
• ILLUMINER: Instruction-tuned Large Language Models as Few-shot Intent Classifier and Slot Filler [1.9015367254988451]
  This study evaluates instruction-tuned models (Instruct-LLMs) on popular benchmark datasets for intent classification (IC) and slot filling (SF) We introduce ILLUMINER, an approach framing IC and SF as language generation tasks for Instruct-LLMs, with a more efficient SF-prompting method compared to prior work. A comprehensive comparison with multiple baselines shows that our approach, using the FLAN-T5 11B model, outperforms the state-of-the-art joint IC+SF method and in-context learning with GPT3.5 (175B).
  arXiv  Detail & Related papers  (2024-03-26T09:41:21Z)
• Japanese-English Sentence Translation Exercises Dataset for Automatic Grading [16.564184260893946]
  This paper proposes the task of automatic assessment of Sentence Translation Exercises (STEs) We create a dataset for STE between Japanese and English including 21 questions, along with a total of 3, 498 student responses (167 on average) Using this dataset, we demonstrate the performance of baselines including finetuned BERT and GPT models with few-shot in-context learning.
  arXiv  Detail & Related papers  (2024-03-06T01:37:03Z)
• Disco-Bench: A Discourse-Aware Evaluation Benchmark for Language Modelling [70.23876429382969]
  We propose a benchmark that can evaluate intra-sentence discourse properties across a diverse set of NLP tasks. Disco-Bench consists of 9 document-level testsets in the literature domain, which contain rich discourse phenomena. For linguistic analysis, we also design a diagnostic test suite that can examine whether the target models learn discourse knowledge.
  arXiv  Detail & Related papers  (2023-07-16T15:18:25Z)
• Pre-Training to Learn in Context [138.0745138788142]
  The ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. We propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters.
  arXiv  Detail & Related papers  (2023-05-16T03:38:06Z)
• Context Matters: A Strategy to Pre-train Language Model for Science Education [4.053049694533914]
  BERT-based language models have shown significant superiority over traditional NLP models in various language-related tasks. The language used by students is different from the language in journals and Wikipedia, which are training sources of BERT. Our study confirms the effectiveness of continual pre-training on domain-specific data in the education domain.
  arXiv  Detail & Related papers  (2023-01-27T23:50:16Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Unifying Language Learning Paradigms [96.35981503087567]
  We present a unified framework for pre-training models that are universally effective across datasets and setups. We show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
  arXiv  Detail & Related papers  (2022-05-10T19:32:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.