LegalWiz: A Multi-Agent Generation Framework for Contradiction Detection in Legal Documents

• URL: http://arxiv.org/abs/2510.03418v2
• Date: Fri, 10 Oct 2025 19:46:31 GMT
• Title: LegalWiz: A Multi-Agent Generation Framework for Contradiction Detection in Legal Documents
• Authors: Ananya Mantravadi, Shivali Dalmia, Olga Pospelova, Abhishek Mukherji, Nand Dave, Anudha Mittal,
• Abstract summary: We present a multi-agent contradiction-aware benchmark framework for the legal domain.<n>It generates synthetic legal-style documents, injects six structured contradiction types, and models both self- and pairwise inconsistencies.<n>This benchmark offers one of the first structured resources for contradiction-aware evaluation in legal RAG pipelines.
• Score: 0.10260880679794955
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-Augmented Generation (RAG) integrates large language models (LLMs) with external sources, but unresolved contradictions in retrieved evidence often lead to hallucinations and legally unsound outputs. Benchmarks currently used for contradiction detection lack domain realism, cover only limited conflict types, and rarely extend beyond single-sentence pairs, making them unsuitable for legal applications. Controlled generation of documents with embedded contradictions is therefore essential: it enables systematic stress-testing of models, ensures coverage of diverse conflict categories, and provides a reliable basis for evaluating contradiction detection and resolution. We present a multi-agent contradiction-aware benchmark framework for the legal domain that generates synthetic legal-style documents, injects six structured contradiction types, and models both self- and pairwise inconsistencies. Automated contradiction mining is combined with human-in-the-loop validation to guarantee plausibility and fidelity. This benchmark offers one of the first structured resources for contradiction-aware evaluation in legal RAG pipelines, supporting more consistent, interpretable, and trustworthy systems.

Related papers

• Diagnosing Knowledge Conflict in Multimodal Long-Chain Reasoning [78.86309644343295]
  Multimodal large language models (MLLMs) in long chain-of-thought reasoning often fail when different knowledge sources provide conflicting signals.<n>We formalize these failures under a unified notion of knowledge conflict, distinguishing input-level objective conflict from process-level effective conflict.<n>Our findings provide a mechanism-level view of multimodal reasoning under knowledge conflict and enable principled diagnosis and control of long-CoT failures.
  arXiv  Detail & Related papers  (2026-02-16T07:10:44Z)
• Adversarial Yet Cooperative: Multi-Perspective Reasoning in Retrieved-Augmented Language Models [72.4149653187766]
  We propose a Reasoner-Verifier framework named Adrialversa Reasoning RAG (ARR)<n>The Reasoner and Verifier engage in reasoning on retrieved evidence and critiquing each other's logic while being guided by process-aware advantage.<n> Experiments on multiple benchmarks demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2026-01-08T06:57:03Z)
• CLASH: A Benchmark for Cross-Modal Contradiction Detection [15.134491772506196]
  CLASH is a novel benchmark for multimodal contradiction detection.<n>It features COCO images paired with contradictory captions containing controlled object-level or attribute-level contradictions.
  arXiv  Detail & Related papers  (2025-11-24T15:09:07Z)
• SeCon-RAG: A Two-Stage Semantic Filtering and Conflict-Free Framework for Trustworthy RAG [35.42029959485188]
  Retrieval-augmented generation (RAG) systems enhance large language models with external knowledge.<n>Existing defenses often apply aggressive filtering, leading to unnecessary loss of valuable information.<n>We propose a two-stage semantic filtering and conflict-free framework for trustworthy RAG.
  arXiv  Detail & Related papers  (2025-10-10T03:44:29Z)
• ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Search [69.60882125603133]
  We present ReliabilityRAG, a framework for adversarial robustness that explicitly leverages reliability information of retrieved documents.<n>Our work is a significant step towards more effective, provably robust defenses against retrieved corpus corruption in RAG.
  arXiv  Detail & Related papers  (2025-09-27T22:36:42Z)
• D-REX: A Benchmark for Detecting Deceptive Reasoning in Large Language Models [62.83226685925107]
  Deceptive Reasoning Exposure Suite (D-REX) is a novel dataset designed to evaluate the discrepancy between a model's internal reasoning process and its final output.<n>Each sample in D-REX contains the adversarial system prompt, an end-user's test query, the model's seemingly innocuous response, and, crucially, the model's internal chain-of-thought.<n>We demonstrate that D-REX presents a significant challenge for existing models and safety mechanisms.
  arXiv  Detail & Related papers  (2025-09-22T15:59:40Z)
• Co-Investigator AI: The Rise of Agentic AI for Smarter, Trustworthy AML Compliance Narratives [2.7295959384567356]
  Co-Investigator AI is an agentic framework optimized to produce Suspicious Activity Reports (SARs) significantly faster and with greater accuracy than traditional methods.<n>We demonstrate its ability to streamline SAR drafting, align narratives with regulatory expectations, and enable compliance teams to focus on higher-order analytical work.
  arXiv  Detail & Related papers  (2025-09-10T08:16:04Z)
• AI-Supported Platform for System Monitoring and Decision-Making in Nuclear Waste Management with Large Language Models [1.6795461001108096]
  This paper presents a multi-agent Retrieval-Augmented Generation (RAG) system that integrates large language models (LLMs) with document retrieval mechanisms.<n>The system ensures regulatory decisions remain factually grounded, dynamically adapting to evolving regulatory frameworks.
  arXiv  Detail & Related papers  (2025-05-27T20:29:53Z)
• AlignRAG: Leveraging Critique Learning for Evidence-Sensitive Retrieval-Augmented Reasoning [61.28113271728859]
  RAG has become a widely adopted paradigm for enabling knowledge-grounded large language models (LLMs)<n>Standard RAG pipelines often fail to ensure that model reasoning remains consistent with the evidence retrieved, leading to factual inconsistencies or unsupported conclusions.<n>In this work, we reinterpret RAG as Retrieval-Augmented Reasoning and identify a central but underexplored problem: textitReasoning Misalignment.
  arXiv  Detail & Related papers  (2025-04-21T04:56:47Z)
• Contradiction Detection in RAG Systems: Evaluating LLMs as Context Validators for Improved Information Consistency [0.6827423171182154]
  Retrieval Augmented Generation (RAG) systems have emerged as a powerful method for enhancing large language models (LLMs) with up-to-date information.<n>RAG can sometimes surface documents containing contradictory information, particularly in rapidly evolving domains such as news.<n>This study presents a novel data generation framework to simulate different types of contradictions that may occur in the retrieval stage of a RAG system.
  arXiv  Detail & Related papers  (2025-03-31T19:41:15Z)
• Introducing Verification Task of Set Consistency with Set-Consistency Energy Networks [4.545178162750511]
  We introduce the task of set-consistency verification, an extension of natural language inference (NLI)<n>We present the Set-Consistency Energy Network (SC-Energy), a novel model that employs a contrastive loss framework to learn the compatibility among a collection of statements.<n>Our approach not only efficiently verifies inconsistencies and pinpoints the specific statements responsible for logical contradictions, but also significantly outperforms existing methods.
  arXiv  Detail & Related papers  (2025-03-12T05:11:11Z)
• Bridging the Gap in XAI-Why Reliable Metrics Matter for Explainability and Compliance [2.3020018305241337]
  The paper emphasizes the critical gap in the evaluation of Explainable AI (XAI) due to the lack of standardized and reliable metrics.<n>Current evaluation methods are often fragmented, subjective, and biased, making them prone to manipulation and complicating the assessment of complex models.<n>We advocate for widespread research into developing robust, context-sensitive evaluation metrics.
  arXiv  Detail & Related papers  (2025-02-07T06:54:48Z)
• SparseCL: Sparse Contrastive Learning for Contradiction Retrieval [87.02936971689817]
  Contradiction retrieval refers to identifying and extracting documents that explicitly disagree with or refute the content of a query. Existing methods such as similarity search and crossencoder models exhibit significant limitations. We introduce SparseCL that leverages specially trained sentence embeddings designed to preserve subtle, contradictory nuances between sentences.
  arXiv  Detail & Related papers  (2024-06-15T21:57:03Z)
• Auditing and Generating Synthetic Data with Controllable Trust Trade-offs [54.262044436203965]
  We introduce a holistic auditing framework that comprehensively evaluates synthetic datasets and AI models. It focuses on preventing bias and discrimination, ensures fidelity to the source data, assesses utility, robustness, and privacy preservation. We demonstrate the framework's effectiveness by auditing various generative models across diverse use cases.
  arXiv  Detail & Related papers  (2023-04-21T09:03:18Z)
• ADC: Adversarial attacks against object Detection that evade Context consistency checks [55.8459119462263]
  We show that even context consistency checks can be brittle to properly crafted adversarial examples. We propose an adaptive framework to generate examples that subvert such defenses. Our results suggest that how to robustly model context and check its consistency, is still an open problem.
  arXiv  Detail & Related papers  (2021-10-24T00:25:09Z)
• Trustworthy Artificial Intelligence and Process Mining: Challenges and Opportunities [0.8602553195689513]
  We show that process mining can provide a useful framework for gaining fact-based visibility to AI compliance process execution. We provide for an automated approach to analyze, remediate and monitor uncertainty in AI regulatory compliance processes.
  arXiv  Detail & Related papers  (2021-10-06T12:50:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.