Adapting Small Language Models to Low-Resource Domains: A Case Study in Hindi Tourism QA

• URL: http://arxiv.org/abs/2510.25273v1
• Date: Wed, 29 Oct 2025 08:32:22 GMT
• Title: Adapting Small Language Models to Low-Resource Domains: A Case Study in Hindi Tourism QA
• Authors: Sandipan Majhi, Paheli Bhattacharya,
• Abstract summary: Domain-specific question answering in low-resource languages faces two key challenges: scarcity of annotated datasets and limited domain knowledge in general-purpose language models.<n>We present a multi-stage finetuning strategy to adapt lightweight language models to the Hindi tourism domain by leveraging both original and synthetic training data.
• Score: 0.509780930114934
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Domain-specific question answering in low-resource languages faces two key challenges: scarcity of annotated datasets and limited domain knowledge in general-purpose language models. In this work, we present a multi-stage finetuning strategy to adapt lightweight language models to the Hindi tourism domain by leveraging both original and synthetic training data. Synthetic question-answer pairs are generated using large LLMs (LLaMA-70B, Phi-14B) and used to augment the limited original dataset. We explore several training methodologies and analyse their impact on domain generalisation. Our results demonstrate that large models can efficiently generate synthetic data, while small models can effectively adapt to it, offering a scalable pathway for low-resource, domain-specific QA.

Related papers

• Using (Not-so) Large Language Models to Generate Simulation Models in a Formal DSL: A Study on Reaction Networks [0.0]
  We evaluate how a Large Language Model might be used for formalizing natural language into simulation models.<n>We develop a synthetic data generator to serve as the basis for fine-tuning and evaluation.<n>Our evaluation shows that our fine-tuned Mistral model can recover the ground truth simulation model in up to 84.5% of cases.
  arXiv  Detail & Related papers  (2025-03-03T15:48:01Z)
• Enhancing Code Generation for Low-Resource Languages: No Silver Bullet [55.39571645315926]
  Large Language Models (LLMs) rely on large and diverse datasets to learn syntax, semantics, and usage patterns of programming languages.<n>For low-resource languages, the limited availability of such data hampers the models' ability to generalize effectively.<n>We present an empirical study investigating the effectiveness of several approaches for boosting LLMs' performance on low-resource languages.
  arXiv  Detail & Related papers  (2025-01-31T12:23:28Z)
• On Domain-Adaptive Post-Training for Multimodal Large Language Models [78.65220510401045]
  This paper systematically investigates domain adaptation of MLLMs via post-training.<n>We focus on data synthesis, training pipeline, and task evaluation.<n>We conduct experiments in high-impact domains such as biomedicine, food, and remote sensing.
  arXiv  Detail & Related papers  (2024-11-29T18:42:28Z)
• Cross-Domain Content Generation with Domain-Specific Small Language Models [3.2772349789781616]
  This study explores methods to enable a small language model to produce coherent and relevant outputs for two different domains. We find that utilizing custom tokenizers tailored to each dataset significantly enhances generation quality. Our findings demonstrate that knowledge expansion with frozen layers is an effective method for small language models to generate domain-specific content.
  arXiv  Detail & Related papers  (2024-09-19T21:45:13Z)
• InkubaLM: A small language model for low-resource African languages [9.426968756845389]
  InkubaLM is a small language model with 0.4 billion parameters. It achieves performance comparable to models with significantly larger parameter counts. It demonstrates remarkable consistency across multiple languages.
  arXiv  Detail & Related papers  (2024-08-30T05:42:31Z)
• Prompting Encoder Models for Zero-Shot Classification: A Cross-Domain Study in Italian [75.94354349994576]
  This paper explores the feasibility of employing smaller, domain-specific encoder LMs alongside prompting techniques to enhance performance in specialized contexts. Our study concentrates on the Italian bureaucratic and legal language, experimenting with both general-purpose and further pre-trained encoder-only models. The results indicate that while further pre-trained models may show diminished robustness in general knowledge, they exhibit superior adaptability for domain-specific tasks, even in a zero-shot setting.
  arXiv  Detail & Related papers  (2024-07-30T08:50:16Z)
• Unlocking the Potential of Model Merging for Low-Resource Languages [66.7716891808697]
  Adapting large language models to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT) We propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. Experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data.
  arXiv  Detail & Related papers  (2024-07-04T15:14:17Z)
• GeMQuAD : Generating Multilingual Question Answering Datasets from Large Language Models using Few Shot Learning [4.8838210812204235]
  In this paper, we propose GeMQuAD - a semi-supervised learning approach, applied to a dataset generated through ICL with just one example in the target language. We iteratively identify high-quality data to enhance model performance, especially for low-resource multilingual setting. Our framework outperforms the machine translation-augmented model by 0.22/1.68 F1/EM points for Hindi and 0.82/1.37 F1/EM points for Spanish on the MLQA dataset.
  arXiv  Detail & Related papers  (2024-04-14T06:55:42Z)
• LLM Augmented LLMs: Expanding Capabilities through Composition [56.40953749310957]
  CALM -- Composition to Augment Language Models -- introduces cross-attention between models to compose their representations and enable new capabilities. We illustrate that augmenting PaLM2-S with a smaller model trained on low-resource languages results in an absolute improvement of up to 13% on tasks like translation into English. When PaLM2-S is augmented with a code-specific model, we see a relative improvement of 40% over the base model for code generation and explanation tasks.
  arXiv  Detail & Related papers  (2024-01-04T18:53:01Z)
• Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models [15.991777903345575]
  Large language models can generalize to novel downstream tasks with relatively few labeled examples. Alternatively, smaller models can solve specific tasks if fine-tuned with enough labeled examples. We study synthetic data generation of fine-tuning training data via fine-tuned teacher LLMs to improve the downstream performance of much smaller models.
  arXiv  Detail & Related papers  (2023-10-02T11:49:05Z)
• Improving Classifier Training Efficiency for Automatic Cyberbullying Detection with Feature Density [58.64907136562178]
  We study the effectiveness of Feature Density (FD) using different linguistically-backed feature preprocessing methods. We hypothesise that estimating dataset complexity allows for the reduction of the number of required experiments. The difference in linguistic complexity of datasets allows us to additionally discuss the efficacy of linguistically-backed word preprocessing.
  arXiv  Detail & Related papers  (2021-11-02T15:48:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.