ImmigrationQA: A Source-Grounded Dataset and Small-Model Adaptation for U.S. Immigration Law

• URL: http://arxiv.org/abs/2605.30589v1
• Date: Thu, 28 May 2026 21:36:18 GMT
• Title: ImmigrationQA: A Source-Grounded Dataset and Small-Model Adaptation for U.S. Immigration Law
• Abstract summary: We describe the construction of ImmigrationQA, a source-grounded question-answering dataset of 17,058 pairs.<n>The corpus was assembled from 11 primary and secondary sources.<n>The fine-tuned model was evaluated against a held-out split of 993 pairs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: U.S. immigration law spans thousands of pages of official policy, federal regulations, and procedural guidance that change frequently and carry high stakes for petitioners who lack legal representation. We describe the construction of ImmigrationQA, a source-grounded question-answering dataset of 17,058 pairs across 13 immigration subdomains, and the fine-tuning of a Llama 3.2 3B Instruct model on that dataset using parameter-efficient LoRA. The corpus was assembled from 11 primary and secondary sources -- including the USCIS Policy Manual, 8 CFR, BIA precedent decisions, and community Q&A -- yielding 10,056 validated canonical documents and 18,308 text chunks. Structured QA pairs were generated from these chunks using Claude Sonnet 4.6 via five mode-specific prompts, with 22 pairs rejected for insufficient source-span overlap. The fine-tuned model was evaluated against a held-out split of 993 pairs using LLM-as-judge scoring on a 101-example stratified sample. The fine-tuned model scored a mean of 1.08/3.0 (16.8% fully correct; 101-example stratified eval) versus the Llama 3 8B base model at 0.85/3.0 (4% fully correct), a relative improvement of 27% in mean score; a zero-shot Claude Sonnet baseline scored 1.52/3.0 (25% fully correct). The fine-tuned model shows concentrated improvement in procedural subdomains (travel documents, adjustment of status, nonimmigrant visas) while remaining weak on complex legal reasoning and time-sensitive statistics. The full pipeline ran for approximately $29 in cloud compute. All artifacts -- dataset, model, code, and prompt templates -- are publicly released. The system is not a substitute for legal counsel and does not reflect regulatory changes after the corpus crawl date.

Related papers

• State of Thought Enables Endogenous Reasoning [48.902117018115256]
  Test-time reasoning has emerged as a major approach to improving the capabilities of Large Language Models (LLMs)<n>We propose State of Thought, a new reasoning paradigm that enables endogenous reasoning in LLMs.<n>We show that SoT consistently improves mean-baseline accuracy while reducing generated tokens by 62.6% and end-to-end latency by 44.6%.
  arXiv  Detail & Related papers  (2026-09-13T05:55:03Z)
• Do Small Models Use the Law You Give Them? Measuring Context Use on a Bilingual Bangladesh Legal Benchmark [2.062723244880136]
  Fine-tuning can improve legal question-answering accuracy without improving how models use law supplied in context.<n>We construct a hierarchy-preserving statutory corpus, 2,165 reviewed bilingual fine-tuning examples, and a 150-item supplied-law control.
  arXiv  Detail & Related papers  (2026-08-31T06:45:16Z)
• Confidently Wrong: Exception Chain Collapse in Frontier LLM Rule Evaluation [0.0]
  We document a failure class in frontier large language models.<n>The failure reproduces at first observation, but its empirical surface is unstable.<n>For regulated, frontier-model accuracy is a moving compliance boundary that shifts without notice.
  arXiv  Detail & Related papers  (2026-07-25T22:40:02Z)
• A Sovereign, Open-Source Foundation Model for German and English [50.08096276509294]
  Soofi S 30B-A3B is a sovereign, open-source Mamba Transformer foundation model for German and English.<n>Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows.
  arXiv  Detail & Related papers  (2026-07-10T13:51:41Z)
• Citation-Closure Retrieval and Per-Rule Attribution for Real-World Regulatory Compliance Question Answering [54.97384993676565]
  We formalize Regulatory Compliance QA with RegOps-Bench, a novel benchmark featuring an Operational Knowledge Graph derived from complex national R&D regulations.<n>RefWalk traverses cross-document citations, fuses multi-view candidates via max-based aggregation, and enforces per-rule attribution to explicitly map claims to sources.
  arXiv  Detail & Related papers  (2026-05-28T10:38:38Z)
• UA-Legal-Bench: A Benchmark for Evaluating Large Language Models on Ukrainian Legal Reasoning [0.0]
  We introduce UA-Legal-Bench, a five-task benchmark for evaluating large language models on Ukrainian legal reasoning.<n>We evaluate 11 LLMs from five families under zero-shot and 3-shot prompting via AWS Bedrock with 158K API calls.<n>Our results reveal sharply task-dependent few-shot effects: few-shot prompting improves judgment form classification by up to +38.6 pp but has mixed effects on outcome prediction.
  arXiv  Detail & Related papers  (2026-05-27T23:12:20Z)
• RefusalBench: Why Refusal Rate Misranks Frontier LLMs on Biological Research Prompts [0.0]
  Strict refusal rates span 0.1% to 94.6% on identical prompts.<n>Nine of 18 frontier models exhibit a hedge-but-help partial-compliance pattern at dual-use tier that binary refusal metrics cannot detect.
  arXiv  Detail & Related papers  (2026-05-20T09:53:31Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Automatic Construction of a Legal Citation Graph from 100 Million Ukrainian Court Decisions: Large-Scale Extraction, Topological Analysis, and Ontology-Driven Clustering [0.0]
  Citation edges extracted from 100.7 million Ukrainian court decisions reveal judicial citation structure encodes legal domain boundaries without supervision.<n>We construct the first large-scale citation graph from the completeSR registry (99.5 million full texts, 1.1 TB)
  arXiv  Detail & Related papers  (2026-05-14T19:42:20Z)
• Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation [0.0]
  We compare tested models to the contemporaneous frontier on the Epoch AI Capabilities Index.<n>The gap between these answers is widening at +5.53 ECI/year.<n>Proposed remedies include API-access subsidies and editorial enforcement of reporting frameworks.
  arXiv  Detail & Related papers  (2026-05-05T17:58:35Z)
• LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification [0.0]
  We introduce LegalBench-BR, the first public benchmark for evaluating language models on Brazilian legal text classification.<n>The dataset comprises 3,105 appellate proceedings from the Santa Catarina State Court (TJSC)<n>On a class-balanced test set, BERTimbau-LoRA achieves 87.6% accuracy and 0.87 macro-F1 (+22pp over Claude 3.5 Haiku, +28pp over GPT-4o mini)
  arXiv  Detail & Related papers  (2026-04-20T22:00:02Z)
• From Black Box to Glass Box: Cross-Model ASR Disagreement to Prioto Review in Ambient AI Scribe Documentation [43.148402136307716]
  Cross-model disagreement among heterogeneous ASR systems can act as a reference-free uncertainty signal.<n>We transcribed 50 publicly available medical audio clips with eight ASR systems spanning commercial APIs and open-source engines.<n>Low-agreement regions were enriched for content disagreements, with the content fraction increasing from 53.9% to 73.9% across quintiles of high-risk mass.
  arXiv  Detail & Related papers  (2026-03-02T13:02:13Z)
• Relative Scaling Laws for LLMs [91.73497548097775]
  Scaling laws describe how language models improve with additional data, parameters, and compute.<n>We introduce relative scaling laws, which track how performance gaps between test distributions evolve with scale.<n>These results show that although scaling improves overall performance, it is not a universal equalizer.
  arXiv  Detail & Related papers  (2025-10-28T16:55:22Z)
• SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models [43.729576865473625]
  Sandwiched Policy Gradient (SPG)<n>Experiments show that SPG significantly outperforms baselines based on ELBO or one-step estimation.<n>Specifically, SPG improves the accuracy over state-of-the-art RL methods for dLLMs by 3.6% in GSM8K, 2.6% in MATH500, 18.4% in Countdown and 27.0% in Sudoku.
  arXiv  Detail & Related papers  (2025-10-10T16:52:25Z)
• The impact of fine tuning in LLaMA on hallucinations for named entity extraction in legal documentation [41.04830252637936]
  The extraction of information about traffic accidents from legal documents is crucial for quantifying insurance company costs.<n>A two-step procedure is proposed: first, segmenting the document identifying the most relevant segments, and then extracting the entities.<n>For text segmentation, two methodologies are compared: a classic method based on regular expressions and a second approach that divides the document into blocks of n-tokens.<n>Large language models (LLaMA-2 7b, 70b, LLaMA-3 8b, and GPT-4 Turbo) are applied with prompting to the selected segments for entity extraction.
  arXiv  Detail & Related papers  (2025-06-10T14:17:12Z)
• An Embarrassingly Simple Defense Against LLM Abliteration Attacks [47.347413305965006]
  A recent attack, termed abliteration, identifies and suppresses the single latent direction most responsible for refusal behavior.<n>We propose a defense that fundamentally alters how models express refusal.<n>Fine-tuning Llama-2-7B-Chat and Qwen2.5-Instruct yields models that maintain high refusal rates under abliteration.
  arXiv  Detail & Related papers  (2025-05-25T09:18:24Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.