Bridging Law and Data: Augmenting Reasoning via a Semi-Structured Dataset with IRAC methodology

• URL: http://arxiv.org/abs/2406.13217v1
• Date: Wed, 19 Jun 2024 04:59:09 GMT
• Title: Bridging Law and Data: Augmenting Reasoning via a Semi-Structured Dataset with IRAC methodology
• Authors: Xiaoxi Kang, Lizhen Qu, Lay-Ki Soon, Zhuang Li, Adnan Trakic,
• Abstract summary: This paper introduces LEGALSEMI, a benchmark specifically curated for legal scenario analysis. LEGALSEMI comprises 54 legal scenarios, each rigorously annotated by legal experts, based on the comprehensive IRAC (Issue, Rule, Application, Conclusion) framework. A series of experiments were conducted to assess the usefulness of LEGALSEMI for IRAC analysis.
• Score: 22.740895683854568
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The effectiveness of Large Language Models (LLMs) in legal reasoning is often limited due to the unique legal terminologies and the necessity for highly specialized knowledge. These limitations highlight the need for high-quality data tailored for complex legal reasoning tasks. This paper introduces LEGALSEMI, a benchmark specifically curated for legal scenario analysis. LEGALSEMI comprises 54 legal scenarios, each rigorously annotated by legal experts, based on the comprehensive IRAC (Issue, Rule, Application, Conclusion) framework. In addition, LEGALSEMI is accompanied by a structured knowledge graph (SKG). A series of experiments were conducted to assess the usefulness of LEGALSEMI for IRAC analysis. The experimental results demonstrate the effectiveness of incorporating the SKG for issue identification, rule retrieval, application and conclusion generation using four different LLMs. LEGALSEMI will be publicly available upon acceptance of this paper.

Related papers

• InternLM-Law: An Open Source Chinese Legal Large Language Model [72.2589401309848]
  InternLM-Law is a specialized LLM tailored for addressing diverse legal queries related to Chinese laws. We meticulously construct a dataset in the Chinese legal domain, encompassing over 1 million queries. InternLM-Law achieves the highest average performance on LawBench, outperforming state-of-the-art models, including GPT-4, on 13 out of 20 subtasks.
  arXiv  Detail & Related papers  (2024-06-21T06:19:03Z)
• BLT: Can Large Language Models Handle Basic Legal Text? [50.46167465931653]
  GPT-4, Claude, and PaLM 2 perform poorly at basic legal text handling. Fine-tuning for these tasks brings even a smaller model to near-perfect performance on our test set.
  arXiv  Detail & Related papers  (2023-11-16T09:09:22Z)
• Using Large Language Models to Support Thematic Analysis in Empirical Legal Studies [0.7673339435080445]
  We propose a novel framework facilitating effective collaboration of a legal expert with a large language model (LLM) We employed the framework for an analysis of a dataset (n=785) of facts descriptions from criminal court opinions regarding thefts.
  arXiv  Detail & Related papers  (2023-10-28T15:20:44Z)
• Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer? [14.103170412148584]
  ChatGPT is applied to perform analysis on the corpus using the IRAC method. Each scenario in the corpus is annotated with a complete IRAC analysis in a semi-structured format. In addition, we conducted the first empirical assessment of ChatGPT for IRAC analysis in order to understand how well it aligns with the analysis of legal professionals.
  arXiv  Detail & Related papers  (2023-10-23T12:51:49Z)
• A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction [60.70089334782383]
  Large language models (LLMs) have demonstrated great potential for domain-specific applications. Recent disputes over GPT-4's law evaluation raise questions concerning their performance in real-world legal tasks. We design practical baseline solutions based on LLMs and test on the task of legal judgment prediction.
  arXiv  Detail & Related papers  (2023-10-18T07:38:04Z)
• Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration [52.57055162778548]
  Legal Judgment Prediction (LJP) has become an increasingly crucial task in Legal AI. Precedents are the previous legal cases with similar facts, which are the basis for the judgment of the subsequent case in national legal systems. Recent advances in deep learning have enabled a variety of techniques to be used to solve the LJP task.
  arXiv  Detail & Related papers  (2023-10-13T16:47:20Z)
• LAiW: A Chinese Legal Large Language Models Benchmark [17.66376880475554]
  General and legal domain LLMs have demonstrated strong performance in various tasks of LegalAI. We are the first to build the Chinese legal LLMs benchmark LAiW, based on the logic of legal practice.
  arXiv  Detail & Related papers  (2023-10-09T11:19:55Z)
• Legal Summarisation through LLMs: The PRODIGIT Project [4.840725842638346]
  PRODIGIT aims to support tax judges and lawyers through digital technology, focusing on AI. We have focused on generation of summaries of judicial decisions and on the extraction of related information. We have deployed and evaluated different tools and approaches to extractive and abstractive summarisation.
  arXiv  Detail & Related papers  (2023-08-04T16:59:48Z)
• Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence [5.07013500385659]
  This paper explores Large Language Models' (LLMs) capabilities in applying tax law. Our experiments demonstrate emerging legal understanding capabilities, with improved performance in each subsequent OpenAI model release. Findings indicate that LLMs, particularly when combined with prompting enhancements and the correct legal texts, can perform at high levels of accuracy but not yet at expert tax lawyer levels.
  arXiv  Detail & Related papers  (2023-06-12T12:40:48Z)
• SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval [75.05173891207214]
  Legal case retrieval plays a core role in the intelligent legal system. Most existing language models have difficulty understanding the long-distance dependencies between different structures. We propose a new Structure-Aware pre-traIned language model for LEgal case Retrieval.
  arXiv  Detail & Related papers  (2023-04-22T10:47:01Z)
• A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges [73.34944216896837]
  Legal judgment prediction (LJP) applies Natural Language Processing (NLP) techniques to predict judgment results based on fact descriptions automatically. We analyze 31 LJP datasets in 6 languages, present their construction process and define a classification method of LJP. We show the state-of-art results for 8 representative datasets from different court cases and discuss the open challenges.
  arXiv  Detail & Related papers  (2022-04-11T04:06:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.