Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh

• URL: http://arxiv.org/abs/2607.23446v1
• Date: Sun, 26 Jul 2026 04:06:13 GMT
• Title: Do Small Models Use the Law You Give Them? Context-Injected Fine-Tuning for Legal QA in Bangladesh
• Abstract summary: A small language model can receive the governing statutory provision and still answer incorrectly.<n>We test whether fine-tuning on examples containing relevant law improves later use of retrieved law.<n> Evaluation uses the 2022 and 2023 Bangladesh Bar Council exams in Bangla and machine-translated English.
• Score: 2.062723244880136
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A small language model can receive the governing statutory provision and still answer incorrectly. We test whether fine-tuning on examples containing relevant law improves later use of retrieved law. We curate 2{,}165 bilingual QA records from six Bangladeshi acts and three schedules, then fine-tune Qwen3.5 at 0.8B, 2B, and 4B. Evaluation uses the 2022 and 2023 Bangladesh Bar Council exams in Bangla and machine-translated English, with no retrieval, BM25, or FAISS, scored by strict consistency over three seeded runs. At 0.8B, fine-tuning raises the 2022 English FAISS score from 2 to 34 of 100. Gains at 0.8B and 2B survive paired testing, but the 4B model has no detectable net gain: Bangla improves while several English conditions regress. Fine-tuning also reduces answers that drift from Bangla into mostly English from 44.0--53.2\% to 0.2--0.7\%, with adjusted $p<.001$ at every scale. Retrieval quality is therefore not the only bottleneck. Small bilingual legal models also differ in how they use supplied law and whether they answer in the requested language. The dataset is publicly available at https://huggingface.co/datasets/momahadi/bangladesh-legal-qa-dataset.

Related papers

• Do Small Models Use the Law You Give Them? Measuring Context Use on a Bilingual Bangladesh Legal Benchmark [2.062723244880136]
  Fine-tuning can improve legal question-answering accuracy without improving how models use law supplied in context.<n>We construct a hierarchy-preserving statutory corpus, 2,165 reviewed bilingual fine-tuning examples, and a 150-item supplied-law control.
  arXiv  Detail & Related papers  (2026-08-31T06:45:16Z)
• Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation [0.0]
  Post-training quantization lowers the memory footprint of Large Language Models (LLMs) and speeds up inference.<n>It is not clear whether the same holds for morphologically complex, low-resource languages such as Bangla.<n>We evaluate three model families---Qwen-2.5-7B, LLaMA-3.1-8B, and GPT-OSS-20B---in full precision and in three quantized formats.
  arXiv  Detail & Related papers  (2026-08-25T14:36:09Z)
• LKValues: Aligning Large Language Models with Sri Lankan Societal Values [41.84116838445288]
  Existing benchmarks overlook Sri Lankan-contextualized values in its official language Sinhala.<n>We propose LKValues, the first survey-grounded resource suite for Sri Lankan value alignment.
  arXiv  Detail & Related papers  (2026-07-22T17:49:37Z)
• GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models [0.06157382820537719]
  The evaluation of mathematical reasoning in large language models (LLMs) has predominantly focused on high-resource languages like English.<n>This study introduces GSM-Plus-BN, a novel perturbated Bengali mathematical dataset.<n>We evaluate six open-source LLMs Qwen3-32B, Llama-3.1-8B-Instant, Llama-3.3-70B-Versatile, Llama-4-Scout-17B-16E-Instruct, GPT-OSS-120B, and GPT-OSS-20B using a benchmark of 9,000 evaluation samples.
  arXiv  Detail & Related papers  (2026-07-14T20:23:27Z)
• A Sovereign, Open-Source Foundation Model for German and English [50.08096276509294]
  Soofi S 30B-A3B is a sovereign, open-source Mamba Transformer foundation model for German and English.<n>Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows.
  arXiv  Detail & Related papers  (2026-07-10T13:51:41Z)
• ImmigrationQA: A Source-Grounded Dataset and Small-Model Adaptation for U.S. Immigration Law [0.0]
  We describe the construction of ImmigrationQA, a source-grounded question-answering dataset of 17,058 pairs.<n>The corpus was assembled from 11 primary and secondary sources.<n>The fine-tuned model was evaluated against a held-out split of 993 pairs.
  arXiv  Detail & Related papers  (2026-05-28T21:36:18Z)
• SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models [0.0]
  We evaluate four open-weight instruction-tuned models on SomaliBench v0.<n>We find large English-to-Somali refusal gaps for all four models.<n>For three models, the dominant Somali non-refusal mode is not fluent harmful compliance but unclear output.
  arXiv  Detail & Related papers  (2026-05-25T04:45:44Z)
• LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification [0.0]
  We introduce LegalBench-BR, the first public benchmark for evaluating language models on Brazilian legal text classification.<n>The dataset comprises 3,105 appellate proceedings from the Santa Catarina State Court (TJSC)<n>On a class-balanced test set, BERTimbau-LoRA achieves 87.6% accuracy and 0.87 macro-F1 (+22pp over Claude 3.5 Haiku, +28pp over GPT-4o mini)
  arXiv  Detail & Related papers  (2026-04-20T22:00:02Z)
• Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning [74.00326879493225]
  We introduce textttSP3F, a framework for enhancing multilingual reasoning without textitany data in the target language(s)<n>First, we supervise fine-tune (SFT) on translated versions of English question-answer pairs to raise base model correctness.<n>Second, we perform RL with feedback from a pairwise judge in a self-play fashion, with the judge receiving the English reference response as textitprivileged information.
  arXiv  Detail & Related papers  (2026-01-26T17:46:44Z)
• Relative Scaling Laws for LLMs [91.73497548097775]
  Scaling laws describe how language models improve with additional data, parameters, and compute.<n>We introduce relative scaling laws, which track how performance gaps between test distributions evolve with scale.<n>These results show that although scaling improves overall performance, it is not a universal equalizer.
  arXiv  Detail & Related papers  (2025-10-28T16:55:22Z)
• Parameter-Efficient Fine-Tuning for Low-Resource Languages: A Comparative Study of LLMs for Bengali Hate Speech Detection [0.0]
  This paper presents the first application of Efficient Fine-Tuning (PEFT) for Bengali hate speech detection using LoRA and QLoRA.<n>Three instruction-tuned large language models were fine-tuned on the BD-SHS dataset of 50,281 annotated comments.<n>Llama-3.2-3B achieved the highest F1-score of 92.23%, followed by Mistral-7B at 88.94% and Gemma-3-4B at 80.25%.
  arXiv  Detail & Related papers  (2025-10-19T20:03:22Z)
• Language models scale reliably with over-training and on downstream tasks [121.69867718185125]
  Scaling laws are useful guides for derisking expensive training runs. However, there remain gaps between current studies and how language models are trained. In contrast, scaling laws mostly predict loss on inference, but models are usually compared on downstream task performance.
  arXiv  Detail & Related papers  (2024-03-13T13:54:00Z)
• How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering [80.82194311274694]
  We examine the question "how can we know when language models know, with confidence, the answer to a particular query?" We examine three strong generative models -- T5, BART, and GPT-2 -- and study whether their probabilities on QA tasks are well calibrated. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness.
  arXiv  Detail & Related papers  (2020-12-02T03:53:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.