Fine-Tuning BERT for Domain-Specific Question Answering: Toward Educational NLP Resources at University Scale

• URL: http://arxiv.org/abs/2512.05179v1
• Date: Thu, 04 Dec 2025 18:27:46 GMT
• Title: Fine-Tuning BERT for Domain-Specific Question Answering: Toward Educational NLP Resources at University Scale
• Abstract summary: The University of Limerick's Department of Electronic and Computer Engineering developed a robot to provide course information to students.<n>A custom dataset of 1,203 question-answer pairs in SQuAD format was constructed using the university book of modules, supplemented with manually and synthetically generated entries.<n>Results show that even modest fine-tuning improves hypothesis framing and knowledge extraction.<n>Our work addresses this gap by showing that fine-tuning BERT with academic pairs yields effective results.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prior work on scientific question answering has largely emphasized chatbot-style systems, with limited exploration of fine-tuning foundation models for domain-specific reasoning. In this study, we developed a chatbot for the University of Limerick's Department of Electronic and Computer Engineering to provide course information to students. A custom dataset of 1,203 question-answer pairs in SQuAD format was constructed using the university book of modules, supplemented with manually and synthetically generated entries. We fine-tuned BERT (Devlin et al., 2019) using PyTorch and evaluated performance with Exact Match and F1 scores. Results show that even modest fine-tuning improves hypothesis framing and knowledge extraction, demonstrating the feasibility of adapting foundation models to educational domains. While domain-specific BERT variants such as BioBERT and SciBERT exist for biomedical and scientific literature, no foundation model has yet been tailored to university course materials. Our work addresses this gap by showing that fine-tuning BERT with academic QA pairs yields effective results, highlighting the potential to scale towards the first domain-specific QA model for universities and enabling autonomous educational knowledge systems.

Related papers

• Curiosity Over Hype: Modeling Motivation Language to Understand Early Outcomes in a Selective Quantum Track [0.0]
  We study whether latent motivation signals in short Spanish admission responses predict engagement and performance in an early quantum computing pathway run by QuantumHub Peru.<n>We analyze applicants' open responses and link them to outcomes from two selective modules.
  arXiv  Detail & Related papers  (2026-02-23T10:09:05Z)
• An Open-Source Dual-Loss Embedding Model for Semantic Retrieval in Higher Education [0.30723404270319693]
  This study presents two open-source embedding models fine-tuned for educational question answering.<n>A synthetic dataset of 3,197 sentence pairs was constructed through a combination of manual curation and large language model (LLM)-assisted generation.<n>Two training strategies were evaluated: (1) a baseline model fine-tuned using MultipleNegativesRankingLoss (MNRL), and (2) a dual-loss model that combines MNRL with CosineSimilarityLoss to improve both semantic ranking and similarity calibration.
  arXiv  Detail & Related papers  (2025-05-08T03:14:14Z)
• Efficient Learning Content Retrieval with Knowledge Injection [0.0]
  We propose a domain-specific application that requires limited resources, utilizing versions of the Phi language model to help learners with educational content.<n>Data required for fine-tuning was obtained from the Huawei Talent Platform.<n>RAG system was used to support the model, which was fine-tuned by 500 Q&A pairs.
  arXiv  Detail & Related papers  (2024-11-28T12:06:14Z)
• Question-Answering (QA) Model for a Personalized Learning Assistant for Arabic Language [4.79071544824946]
  This paper describes the creation, optimization, and assessment of a question-answering (QA) model for a personalized learning assistant that uses BERT transformers customized for the Arabic language. Our approach uses BERT's brilliant capabilities to automatically produce correct answers to questions in the field of science education. The model's ability to understand and extract pertinent information is improved by finetuning it using 11th and 12th grade biology book in Palestinian curriculum.
  arXiv  Detail & Related papers  (2024-06-11T20:23:31Z)
• Language Models as Science Tutors [79.73256703631492]
  We introduce TutorEval and TutorChat to measure real-life usability of LMs as scientific assistants. We show that fine-tuning base models with existing dialogue datasets leads to poor performance on TutorEval. We use TutorChat to fine-tune Llemma models with 7B and 34B parameters. These LM tutors specialized in math have a 32K-token context window, and they excel at TutorEval while performing strongly on GSM8K and MATH.
  arXiv  Detail & Related papers  (2024-02-16T22:24:13Z)
• Towards Efficient Methods in Medical Question Answering using Knowledge Graph Embeddings [3.944219308229571]
  In Natural Language Processing (NLP), Machine Reading (MRC) is the task of answering a question based on a given context.<n>To handle questions in the medical domain, modern language models such as BioBERT, SciBERT and even ChatGPT are trained on vast amounts of in-domain medical corpora.<n>We propose a resource-efficient approach for injecting domain knowledge into a model without relying on such domain-specific pre-training.
  arXiv  Detail & Related papers  (2024-01-15T21:43:46Z)
• Context Matters: A Strategy to Pre-train Language Model for Science Education [4.053049694533914]
  BERT-based language models have shown significant superiority over traditional NLP models in various language-related tasks. The language used by students is different from the language in journals and Wikipedia, which are training sources of BERT. Our study confirms the effectiveness of continual pre-training on domain-specific data in the education domain.
  arXiv  Detail & Related papers  (2023-01-27T23:50:16Z)
• Sparse*BERT: Sparse Models Generalize To New tasks and Domains [79.42527716035879]
  This paper studies how models pruned using Gradual Unstructured Magnitude Pruning can transfer between domains and tasks. We demonstrate that our general sparse model Sparse*BERT can become SparseBioBERT simply by pretraining the compressed architecture on unstructured biomedical text.
  arXiv  Detail & Related papers  (2022-05-25T02:51:12Z)
• When BERT Meets Quantum Temporal Convolution Learning for Text Classification in Heterogeneous Computing [75.75419308975746]
  This work proposes a vertical federated learning architecture based on variational quantum circuits to demonstrate the competitive performance of a quantum-enhanced pre-trained BERT model for text classification. Our experiments on intent classification show that our proposed BERT-QTC model attains competitive experimental results in the Snips and ATIS spoken language datasets.
  arXiv  Detail & Related papers  (2022-02-17T09:55:21Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing [73.37262264915739]
  We show that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks.
  arXiv  Detail & Related papers  (2020-07-31T00:04:15Z)
• Evaluation Toolkit For Robustness Testing Of Automatic Essay Scoring Systems [64.4896118325552]
  We evaluate the current state-of-the-art AES models using a model adversarial evaluation scheme and associated metrics. We find that AES models are highly overstable. Even heavy modifications(as much as 25%) with content unrelated to the topic of the questions do not decrease the score produced by the models.
  arXiv  Detail & Related papers  (2020-07-14T03:49:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.