Enhancing Legal LLMs through Metadata-Enriched RAG Pipelines and Direct Preference Optimization

• URL: http://arxiv.org/abs/2603.19251v1
• Date: Wed, 25 Feb 2026 06:39:37 GMT
• Title: Enhancing Legal LLMs through Metadata-Enriched RAG Pipelines and Direct Preference Optimization
• Abstract summary: Large Language Models (LLMs) perform well in short contexts but degrade on long legal documents.<n>In the legal domain, where precision is critical, such errors undermine reliability and trust.<n>We identify two failure modes: retrieval errors due to lexical redundancy in legal corpora, and decoding errors where models generate answers despite insufficient context.
• Score: 0.3823356975862005
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) perform well in short contexts but degrade on long legal documents, often producing hallucinations such as incorrect clauses or precedents. In the legal domain, where precision is critical, such errors undermine reliability and trust. Retrieval Augmented Generation (RAG) helps ground outputs but remains limited in legal settings, especially with small, locally deployed models required for data privacy. We identify two failure modes: retrieval errors due to lexical redundancy in legal corpora, and decoding errors where models generate answers despite insufficient context. To address this, we propose Metadata Enriched Hybrid RAG to improve document level retrieval, and apply Direct Preference Optimization (DPO) to enforce safe refusal when context is inadequate. Together, these methods improve grounding, reliability, and safety in legal language models.

Related papers

• Building Legal Reward Models for Grounding and Abstention [19.598261101266463]
  We introduce a framework for transforming existing legal QA datasets into contextual preference data.<n>We find that contextual DPO improves grounded evaluation, but performance is sensitive to preference-data construction.<n>We also find evidence of cross-jurisdiction transfer: models contextually refined primarily on Victorian criminal-law data improve grounded evaluation on external US legal benchmarks.
  arXiv  Detail & Related papers  (2026-09-13T18:59:58Z)
• RH-RAG: Trustworthy Long-Form Generation for Privacy-Constrained Settings [0.0]
  RH-RAG is a framework for secure and trustworthy long form generation using local language models.<n>It decomposes generation into three coordinated stages: a Planner Agent that constructs a global document outline from semantic summaries, a Writer Agent that incrementally generates coherent section-wise content, and a Checker Agent that mitigates hallucinations through natural language inference-based factual verification.<n> Evaluations across literary, financial, and legal domains demonstrate that RH-RAG consistently improves factual grounding, semantic coherence, and document-level alignment.
  arXiv  Detail & Related papers  (2026-08-02T15:27:00Z)
• Structure-Aware RAG: Structured Retrieval Augmented Generation from Noisy Data for Conversational Agents [68.52563127206808]
  Structure-aware Retrieval Augmented Generation (SA-RAG) uses tables as an intermediate structured representation to provide a compact and controllable interface.<n>We introduce a quality-aware table metadata generation framework that models metadata normalization and effectiveness.<n>Experiments on two noisy real-world datasets show that SA-RAG significantly outperforms existing RAG baselines.
  arXiv  Detail & Related papers  (2026-05-23T03:07:33Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Adaptive Conformal Prediction for Improving Factuality of Generations by Large Language Models [86.8650252164764]
  Large language models (LLMs) are prone to generating factually incorrect outputs.<n>We propose an adaptive conformal prediction approach that extends conformal score transformation methods to LLMs.<n>This enables prompt-dependent calibration, retaining marginal coverage guarantees while improving conditional coverage.
  arXiv  Detail & Related papers  (2026-04-15T15:35:42Z)
• Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model [99.16364381244445]
  Modern language models (LMs) tend to memorize portions of their training data and emit verbatim spans.<n>We propose Anchored Decoding, a plug-and-play inference-time method for suppressing verbatim copying.<n>We evaluate our methods across six model pairs on long-form evaluations of copyright risk and utility.
  arXiv  Detail & Related papers  (2026-02-06T19:00:14Z)
• Towards Reliable Retrieval in RAG Systems for Large Legal Datasets [6.376251215279889]
  Retrieval-Augmented Generation (RAG) is a promising approach to mitigate hallucinations in Large Language Models (LLMs)<n>This is particularly challenging in the legal domain, where large databases of structurally similar documents often cause retrieval systems to fail.<n>We investigate a simple and computationally efficient technique which enhances each text chunk with a document-level synthetic summary.<n>Our work provides evidence that this practical, scalable, and easily integrable technique enhances the reliability of RAG systems when applied to large-scale legal document datasets.
  arXiv  Detail & Related papers  (2025-10-08T13:22:20Z)
• ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Search [69.60882125603133]
  We present ReliabilityRAG, a framework for adversarial robustness that explicitly leverages reliability information of retrieved documents.<n>Our work is a significant step towards more effective, provably robust defenses against retrieved corpus corruption in RAG.
  arXiv  Detail & Related papers  (2025-09-27T22:36:42Z)
• Nine Ways to Break Copyright Law and Why Our LLM Won't: A Fair Use Aligned Generation Framework [7.941114118462577]
  Large language models (LLMs) commonly risk copyright infringement by reproducing protected content verbatim or with insufficient transformative modifications.<n>We develop a legally-grounded framework explicitly designed to align LLM outputs with fair-use doctrine.<n>FuA-LLM substantially reduces problematic outputs (up to 20%) compared to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-05-25T12:23:26Z)
• Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? [83.53005932513155]
  Multi-modal large language models (MLLMs) have made significant progress, yet their safety alignment remains limited.<n>We propose finetuning MLLMs on a small set of benign instruct-following data with responses replaced by simple, clear rejection sentences.
  arXiv  Detail & Related papers  (2025-04-14T09:03:51Z)
• Reliable Decision Making via Calibration Oriented Retrieval Augmented Generation [30.830648083015607]
  When Large Language Models confidently provide incorrect information, it can lead humans to make suboptimal decisions.<n>We propose Calibrated Retrieval-Augmented Generation (CalibRAG), which ensures that decisions informed by RAG are well-calibrated.
  arXiv  Detail & Related papers  (2024-10-28T06:41:05Z)
• Multi-Reference Preference Optimization for Large Language Models [56.84730239046117]
  We introduce a novel closed-form formulation for direct preference optimization using multiple reference models. The resulting algorithm, Multi-Reference Preference Optimization (MRPO), leverages broader prior knowledge from diverse reference models. Our experiments demonstrate that LLMs finetuned with MRPO generalize better in various preference data, regardless of data scarcity or abundance.
  arXiv  Detail & Related papers  (2024-05-26T00:29:04Z)
• SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore [159.21914121143885]
  We present SILO, a new language model that manages this risk-performance tradeoff during inference. SILO is built by (1) training a parametric LM on Open License Corpus (OLC), a new corpus we curate with 228B tokens of public domain and permissively licensed text. Access to the datastore greatly improves out of domain performance, closing 90% of the performance gap with an LM trained on the Pile.
  arXiv  Detail & Related papers  (2023-08-08T17:58:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.