Towards Trustworthy Legal AI through LLM Agents and Formal Reasoning

• URL: http://arxiv.org/abs/2511.21033v1
• Date: Wed, 26 Nov 2025 04:05:06 GMT
• Title: Towards Trustworthy Legal AI through LLM Agents and Formal Reasoning
• Authors: Linze Chen, Yufan Cai, Zhe Hou, Jinsong Dong,
• Abstract summary: Existing LLM-based systems excel at surface-level text analysis but lack the guarantees required for principled rationality.<n>We introduce L4M, a novel framework that combines LLM agents with SMT-solver-backed jurisprudence.<n>We show that our system surpasses advanced LLMs including GPT-o4-mini, DeepSeek-V3, and Claude 4 as well as state-of-the-art Legal AI proofs.
• Score: 11.842866992683158
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rationality of law manifests in two forms: substantive rationality, which concerns the fairness or moral desirability of outcomes, and formal rationality, which requires legal decisions to follow explicitly stated, general, and logically coherent rules. Existing LLM-based systems excel at surface-level text analysis but lack the guarantees required for principled jurisprudence. We introduce L4M, a novel framework that combines adversarial LLM agents with SMT-solver-backed proofs to unite the interpretive flexibility of natural language with the rigor of symbolic verification. The pipeline consists of three phases: (1) Statute Formalization, where domain-specific prompts convert legal provisions into logical formulae; (2) Dual Fact and Statute Extraction, in which prosecutor- and defense-aligned LLMs independently map case narratives to fact tuples and statutes, ensuring role isolation; and (3) Solver-Centric Adjudication, where an autoformalizer compiles both parties' arguments into logic constraints, and unsat cores trigger iterative self-critique until a satisfiable formula is achieved, which is then verbalized by a Judge-LLM into a transparent verdict and optimized sentence. Experimental results on public benchmarks show that our system surpasses advanced LLMs including GPT-o4-mini, DeepSeek-V3, and Claude 4 as well as state-of-the-art Legal AI baselines, while providing rigorous and explainable symbolic justifications.

Related papers

• LegalOne: A Family of Foundation Models for Reliable Legal Reasoning [54.57434222018289]
  We present LegalOne, a family of foundational models specifically tailored for the Chinese legal domain.<n>LegalOne is developed through a comprehensive three-phase pipeline designed to master legal reasoning.<n>We publicly release the LegalOne weights and the LegalKit evaluation framework to advance the field of Legal AI.
  arXiv  Detail & Related papers  (2026-01-31T10:18:32Z)
• Structured Decomposition for LLM Reasoning: Cross-Domain Validation and Semantic Web Integration [0.0]
  Rule-based reasoning arises in domains where decisions must be auditable and justifiable.<n>Applying rules to such inputs demands both interpretive flexibility and formal guarantees.<n>This paper presents an integration pattern that combines these strengths.
  arXiv  Detail & Related papers  (2026-01-04T17:19:20Z)
• On Verifiable Legal Reasoning: A Multi-Agent Framework with Formalized Knowledge Representations [0.0]
  This paper introduces a modular multi-agent framework that decomposes legal reasoning into distinct knowledge acquisition and application stages.<n>In the first stage, specialized agents extract legal concepts and formalize rules to create verifiable intermediate representations of statutes.<n>The second stage applies this knowledge to specific cases through three steps: analyzing queries to map case facts onto the schema, performing symbolic inference to derive logically entailed conclusions, and generating final answers.
  arXiv  Detail & Related papers  (2025-08-31T06:03:00Z)
• Explainable Rule Application via Structured Prompting: A Neural-Symbolic Approach [0.0]
  Large Language Models (LLMs) excel in complex reasoning tasks but struggle with consistent rule application, exception handling, and explainability.<n>This paper introduces a structured prompting framework that decomposes reasoning into three verifiable steps: entity identification, property extraction, and symbolic rule application.
  arXiv  Detail & Related papers  (2025-06-19T14:14:01Z)
• CLATTER: Comprehensive Entailment Reasoning for Hallucination Detection [60.98964268961243]
  We propose that guiding models to perform a systematic and comprehensive reasoning process allows models to execute much finer-grained and accurate entailment decisions.<n>We define a 3-step reasoning process, consisting of (i) claim decomposition, (ii) sub-claim attribution and entailment classification, and (iii) aggregated classification, showing that such guided reasoning indeed yields improved hallucination detection.
  arXiv  Detail & Related papers  (2025-06-05T17:02:52Z)
• RLJP: Legal Judgment Prediction via First-Order Logic Rule-enhanced with Large Language Models [58.69183479148083]
  Legal Judgment Prediction (LJP) is a pivotal task in legal AI.<n>Existing LJP models integrate judicial precedents and legal knowledge for high performance.<n>But they neglect legal reasoning logic, a critical component of legal judgments requiring rigorous logical analysis.<n>This paper proposes a rule-enhanced legal judgment prediction framework based on first-order logic (FOL) formalism and comparative learning (CL)
  arXiv  Detail & Related papers  (2025-05-27T14:50:21Z)
• Learning to Reason via Mixture-of-Thought for Logical Reasoning [56.24256916896427]
  Mixture-of-Thought (MoT) is a framework that enables LLMs to reason across three complementary modalities: natural language, code, and truth-table.<n>MoT adopts a two-phase design: (1) self-evolving MoT training, which jointly learns from filtered, self-generated rationales across modalities; and (2) MoT inference, which fully leverages the synergy of three modalities to produce better predictions.
  arXiv  Detail & Related papers  (2025-05-21T17:59:54Z)
• An Explicit Syllogistic Legal Reasoning Framework for Large Language Models [5.501226256903341]
  Large language models (LLMs) can answer legal questions, but often struggle with explicit syllogistic reasoning.<n>We introduce SyLeR, a novel framework designed to enable LLMs to perform explicit syllogistic legal reasoning.<n>SyLeR employs a tree-structured hierarchical retrieval mechanism to synthesize relevant legal statutes and precedents.
  arXiv  Detail & Related papers  (2025-04-05T03:34:51Z)
• A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences [76.73731245899454]
  We propose a transparent law reasoning schema enriched with hierarchical factum probandum, evidence, and implicit experience.<n>Inspired by this schema, we introduce the challenging task, which takes a textual case description and outputs a hierarchical structure justifying the final decision.<n>This benchmark paves the way for transparent and accountable AI-assisted law reasoning in the Intelligent Court''
  arXiv  Detail & Related papers  (2025-03-02T10:26:54Z)
• Logical Lease Litigation: Prolog and LLMs for Rental Law Compliance in New York [0.30693357740321775]
  This paper presents a novel approach and system, LogicLease, to automate the analysis of landlord-tenant legal cases in the state of New York.<n> LogicLease determines compliance with relevant legal requirements by analyzing case descriptions and citing all relevant laws.<n>We evaluate the accuracy, efficiency, and robustness of LogicLease through a series of tests, achieving 100% accuracy and an average processing time of 2.57 seconds.
  arXiv  Detail & Related papers  (2025-02-13T11:45:38Z)
• RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios [58.90106984375913]
  RuleArena is a novel and challenging benchmark designed to evaluate the ability of large language models (LLMs) to follow complex, real-world rules in reasoning.<n> Covering three practical domains -- airline baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs' proficiency in handling intricate natural language instructions.
  arXiv  Detail & Related papers  (2024-12-12T06:08:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.