AI and the Law: Evaluating ChatGPT's Performance in Legal Classification

• URL: http://arxiv.org/abs/2502.12193v1
• Date: Sat, 15 Feb 2025 09:28:52 GMT
• Title: AI and the Law: Evaluating ChatGPT's Performance in Legal Classification
• Authors: Pawel Weichbroth,
• Abstract summary: The use of ChatGPT to analyze and classify evidence in criminal proceedings has been a topic of ongoing discussion.<n>This study addresses this research gap by evaluating the effectiveness of ChatGPT in classifying legal cases under the Polish Penal Code.<n>The results show excellent binary classification accuracy, with all positive and negative cases correctly categorized.
• Score: 2.6107298043931197
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The use of ChatGPT to analyze and classify evidence in criminal proceedings has been a topic of ongoing discussion. However, to the best of our knowledge, this issue has not been studied in the context of the Polish language. This study addresses this research gap by evaluating the effectiveness of ChatGPT in classifying legal cases under the Polish Penal Code. The results show excellent binary classification accuracy, with all positive and negative cases correctly categorized. In addition, a qualitative evaluation confirms that the legal basis provided for each case, along with the relevant legal content, was appropriate. The results obtained suggest that ChatGPT can effectively analyze and classify evidence while applying the appropriate legal rules. In conclusion, ChatGPT has the potential to assist interested parties in the analysis of evidence and serve as a valuable legal resource for individuals with less experience or knowledge in this area.

Related papers

• LegalBench.PT: A Benchmark for Portuguese Law [17.554201334646056]
  We present LegalBench.PT, the first comprehensive legal benchmark covering key areas of Portuguese law. We first collect long-form questions and answers from real law exams, and then use GPT-4o to convert them into multiple-choice, true/false, and matching formats.
  arXiv  Detail & Related papers  (2025-02-22T21:07:12Z)
• Learning Interpretable Legal Case Retrieval via Knowledge-Guided Case Reformulation [22.85652668826498]
  This paper introduces KELLER, a legal knowledge-guided case reformulation approach based on large language models (LLMs) By incorporating professional legal knowledge about crimes and law articles, we enable large language models to accurately reformulate the original legal case into concise sub-facts of crimes.
  arXiv  Detail & Related papers  (2024-06-28T08:59:45Z)
• Topic Classification of Case Law Using a Large Language Model and a New Taxonomy for UK Law: AI Insights into Summary Judgment [0.0]
  This paper develops and applies a novel taxonomy for topic classification of summary judgment cases in the United Kingdom.<n>We use the Large Language Model Claude 3 Opus to explore functional topics and trends.<n>We find that Claude 3 Opus correctly classified the topic with an accuracy of 87.13% and an F1 score of 0.87.
  arXiv  Detail & Related papers  (2024-05-21T16:30:25Z)
• DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment [55.91429725404988]
  We introduce DELTA, a discriminative model designed for legal case retrieval. We leverage shallow decoders to create information bottlenecks, aiming to enhance the representation ability. Our approach can outperform existing state-of-the-art methods in legal case retrieval.
  arXiv  Detail & Related papers  (2024-03-27T10:40:14Z)
• Combining topic modelling and citation network analysis to study case law from the European Court on Human Rights on the right to respect for private and family life [0.0]
  This paper focuses on case law from the European Court of Human Rights on Article 8 of the European Convention of Human Rights. We demonstrate and compare the potential of topic modelling and citation network to find and organize case law on Article 8. We evaluate the effectiveness of the combined method on a manually collected and annotated dataset of Aricle 8 case law on evictions.
  arXiv  Detail & Related papers  (2024-01-19T14:30:35Z)
• Large Language Models in Cryptocurrency Securities Cases: Can a GPT Model Meaningfully Assist Lawyers? [0.3441021278275805]
  We study GPT-3.5's legal reasoning and ChatGPT's legal drafting capabilities. We feed fact patterns from real-life cases to GPT-3.5 and evaluate its ability to determine correct potential violations. Second, we had mock jurors assess complaints written by ChatGPT and lawyers.
  arXiv  Detail & Related papers  (2023-08-11T09:23:11Z)
• SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval [75.05173891207214]
  Legal case retrieval plays a core role in the intelligent legal system. Most existing language models have difficulty understanding the long-distance dependencies between different structures. We propose a new Structure-Aware pre-traIned language model for LEgal case Retrieval.
  arXiv  Detail & Related papers  (2023-04-22T10:47:01Z)
• To ChatGPT, or not to ChatGPT: That is the question! [78.407861566006]
  This study provides a comprehensive and contemporary assessment of the most recent techniques in ChatGPT detection. We have curated a benchmark dataset consisting of prompts from ChatGPT and humans, including diverse questions from medical, open Q&A, and finance domains. Our evaluation results demonstrate that none of the existing methods can effectively detect ChatGPT-generated content.
  arXiv  Detail & Related papers  (2023-04-04T03:04:28Z)
• Consistency Analysis of ChatGPT [65.268245109828]
  This paper investigates the trustworthiness of ChatGPT and GPT-4 regarding logically consistent behaviour. Our findings suggest that while both models appear to show an enhanced language understanding and reasoning ability, they still frequently fall short of generating logically consistent predictions.
  arXiv  Detail & Related papers  (2023-03-11T01:19:01Z)
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study [121.77986688862302]
  We provide a preliminary meta-evaluation on ChatGPT to show its reliability as an NLG metric. Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments. We hope our preliminary study could prompt the emergence of a general-purposed reliable NLG metric.
  arXiv  Detail & Related papers  (2023-03-07T16:57:20Z)
• Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT [103.57103957631067]
  ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries. We evaluate ChatGPT's understanding ability by evaluating it on the most popular GLUE benchmark, and comparing it with 4 representative fine-tuned BERT-style models. We find that: 1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT achieves comparable performance compared with BERT on sentiment analysis and question answering tasks.
  arXiv  Detail & Related papers  (2023-02-19T12:29:33Z)
• Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction [46.71918729837462]
  Given the fact description text of a legal case, legal judgment prediction aims to predict the case's charge, law article and penalty term. Previous studies fail to distinguish different classification errors with a standard cross-entropy classification loss. We propose a moco-based supervised contrastive learning to learn distinguishable representations. We further enhance the representation of the fact description with extracted crime amounts which are encoded by a pre-trained numeracy model.
  arXiv  Detail & Related papers  (2022-11-15T15:53:56Z)
• Legal Element-oriented Modeling with Multi-view Contrastive Learning for Legal Case Retrieval [3.909749182759558]
  We propose an interaction-focused network for legal case retrieval with a multi-view contrastive learning objective. Case-view contrastive learning minimizes the hidden space distance between relevant legal case representations. We employ a legal element knowledge-aware indicator to detect legal elements of cases.
  arXiv  Detail & Related papers  (2022-10-11T06:47:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.