Towards Robust Legal Reasoning: Harnessing Logical LLMs in Law

• URL: http://arxiv.org/abs/2502.17638v1
• Date: Mon, 24 Feb 2025 20:38:17 GMT
• Title: Towards Robust Legal Reasoning: Harnessing Logical LLMs in Law
• Authors: Manuj Kant, Sareh Nabi, Manav Kant, Roland Scharrer, Megan Ma, Marzieh Nabi,
• Abstract summary: Large language models (LLMs) show promise, but their application in legal contexts demands higher accuracy, repeatability, and transparency.<n>We propose a neuro-symbolic approach that integrates LLMs' natural language understanding with logic-based reasoning.<n>As a legal document case study, we applied neuro-symbolic AI to coverage-related queries in insurance contracts.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Legal services rely heavily on text processing. While large language models (LLMs) show promise, their application in legal contexts demands higher accuracy, repeatability, and transparency. Logic programs, by encoding legal concepts as structured rules and facts, offer reliable automation, but require sophisticated text extraction. We propose a neuro-symbolic approach that integrates LLMs' natural language understanding with logic-based reasoning to address these limitations. As a legal document case study, we applied neuro-symbolic AI to coverage-related queries in insurance contracts using both closed and open-source LLMs. While LLMs have improved in legal reasoning, they still lack the accuracy and consistency required for complex contract analysis. In our analysis, we tested three methodologies to evaluate whether a specific claim is covered under a contract: a vanilla LLM, an unguided approach that leverages LLMs to encode both the contract and the claim, and a guided approach that uses a framework for the LLM to encode the contract. We demonstrated the promising capabilities of LLM + Logic in the guided approach.

Related papers

• LightPROF: A Lightweight Reasoning Framework for Large Language Model on Knowledge Graph [57.382255728234064]
  Large Language Models (LLMs) have impressive capabilities in text understanding and zero-shot reasoning. Knowledge Graphs (KGs) provide rich and reliable contextual information for the reasoning process of LLMs. We propose a novel Lightweight and efficient Prompt learning-ReasOning Framework for KGQA (LightPROF)
  arXiv  Detail & Related papers  (2025-04-04T03:03:47Z)
• Logical Lease Litigation: Prolog and LLMs for Rental Law Compliance in New York [0.30693357740321775]
  This paper presents a novel approach and system, LogicLease, to automate the analysis of landlord-tenant legal cases in the state of New York.<n> LogicLease determines compliance with relevant legal requirements by analyzing case descriptions and citing all relevant laws.<n>We evaluate the accuracy, efficiency, and robustness of LogicLease through a series of tests, achieving 100% accuracy and an average processing time of 2.57 seconds.
  arXiv  Detail & Related papers  (2025-02-13T11:45:38Z)
• CRANE: Reasoning with constrained LLM generation [5.971462597321995]
  We propose a reasoning-augmented constrained decoding algorithm, CRANE, which balances correctness of constrained generation with flexibility of unconstrained generation.<n> CRANE significantly outperforms both state-of-the-art constrained decoding strategies and standard unconstrained decoding.
  arXiv  Detail & Related papers  (2025-02-13T08:23:42Z)
• Logical Consistency of Large Language Models in Fact-checking [6.286017217366497]
  Large language models (LLMs) have demonstrated significant success in performing varied natural language tasks.<n>Despite their impressive ability to generate human-like texts, LLMs are infamous for their inconsistent responses.
  arXiv  Detail & Related papers  (2024-12-20T17:42:25Z)
• RuAG: Learned-rule-augmented Generation for Large Language Models [62.64389390179651]
  We propose a novel framework, RuAG, to automatically distill large volumes of offline data into interpretable first-order logic rules. We evaluate our framework on public and private industrial tasks, including natural language processing, time-series, decision-making, and industrial tasks.
  arXiv  Detail & Related papers  (2024-11-04T00:01:34Z)
• Caveat Lector: Large Language Models in Legal Practice [0.0]
  The fascination with large language models derives from the fact that many users lack the expertise to evaluate the quality of the generated text. The dangerous combination of fluency and superficial plausibility leads to the temptation to trust the generated text and creates the risk of overreliance.
  arXiv  Detail & Related papers  (2024-03-14T08:19:41Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• Neuro-Symbolic Integration Brings Causal and Reliable Reasoning Proofs [95.07757789781213]
  Two lines of approaches are adopted for complex reasoning with LLMs.<n>One line of work prompts LLMs with various reasoning structures, while the structural outputs can be naturally regarded as intermediate reasoning steps.<n>The other line of work adopt LLM-free declarative solvers to do the reasoning task, rendering higher reasoning accuracy but lacking interpretability due to the black-box nature of the solvers.<n>We present a simple extension to the latter line of work. Specifically, we showcase that the intermediate search logs generated by Prolog interpreters can be accessed and interpreted into human-readable reasoning.
  arXiv  Detail & Related papers  (2023-11-16T11:26:21Z)
• BLT: Can Large Language Models Handle Basic Legal Text? [44.89873147675516]
  GPT-4 and Claude perform poorly on basic legal text handling. Poor performance on benchmark casts into doubt their reliability as-is for legal practice. Fine-tuning on training set brings even a small model to near-perfect performance.
  arXiv  Detail & Related papers  (2023-11-16T09:09:22Z)
• A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction [60.70089334782383]
  Large language models (LLMs) have demonstrated great potential for domain-specific applications. Recent disputes over GPT-4's law evaluation raise questions concerning their performance in real-world legal tasks. We design practical baseline solutions based on LLMs and test on the task of legal judgment prediction.
  arXiv  Detail & Related papers  (2023-10-18T07:38:04Z)
• LawBench: Benchmarking Legal Knowledge of Large Language Models [35.2812008533622]
  Large language models (LLMs) have demonstrated strong capabilities in various aspects. It is unclear how much legal knowledge they possess and whether they can reliably perform legal-related tasks. LawBench has been meticulously crafted to have precise assessment of the LLMs' legal capabilities from three cognitive levels.
  arXiv  Detail & Related papers  (2023-09-28T09:35:59Z)
• Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning [101.26814728062065]
  Large Language Models (LLMs) have shown human-like reasoning abilities but still struggle with complex logical problems. This paper introduces a novel framework, Logic-LM, which integrates LLMs with symbolic solvers to improve logical problem-solving.
  arXiv  Detail & Related papers  (2023-05-20T22:25:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.