Automatic explanation of the classification of Spanish legal judgments in jurisdiction-dependent law categories with tree estimators

• URL: http://arxiv.org/abs/2404.00437v1
• Date: Sat, 30 Mar 2024 17:59:43 GMT
• Title: Automatic explanation of the classification of Spanish legal judgments in jurisdiction-dependent law categories with tree estimators
• Authors: Jaime González-González, Francisco de Arriba-Pérez, Silvia García-Méndez, Andrea Busto-Castiñeira, Francisco J. González-Castaño,
• Abstract summary: This work contributes with a system combining Natural Language Processing (NLP) with Machine Learning (ML) to classify legal texts in an explainable manner. We analyze the features involved in the decision and the threshold bifurcation values of the decision paths of tree structures. Legal experts have validated our solution, and this knowledge has also been incorporated into the explanation process as "expert-in-the-loop" dictionaries.
• Score: 6.354358255072839
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Automatic legal text classification systems have been proposed in the literature to address knowledge extraction from judgments and detect their aspects. However, most of these systems are black boxes even when their models are interpretable. This may raise concerns about their trustworthiness. Accordingly, this work contributes with a system combining Natural Language Processing (NLP) with Machine Learning (ML) to classify legal texts in an explainable manner. We analyze the features involved in the decision and the threshold bifurcation values of the decision paths of tree structures and present this information to the users in natural language. This is the first work on automatic analysis of legal texts combining NLP and ML along with Explainable Artificial Intelligence techniques to automatically make the models' decisions understandable to end users. Furthermore, legal experts have validated our solution, and this knowledge has also been incorporated into the explanation process as "expert-in-the-loop" dictionaries. Experimental results on an annotated data set in law categories by jurisdiction demonstrate that our system yields competitive classification performance, with accuracy values well above 90%, and that its automatic explanations are easily understandable even to non-expert users.

Related papers

• Data2Concept2Text: An Explainable Multilingual Framework for Data Analysis Narration [42.95840730800478]
  This paper presents a complete explainable system that interprets a set of data, abstracts the underlying features and describes them in a natural language of choice. The system relies on two crucial stages: (i) identifying emerging properties from data and transforming them into abstract concepts, and (ii) converting these concepts into natural language.
  arXiv  Detail & Related papers  (2025-02-13T11:49:48Z)
• A Multi-Source Heterogeneous Knowledge Injected Prompt Learning Method for Legal Charge Prediction [3.52209555388364]
  We propose a prompt learning framework-based method for modeling case descriptions. We leverage multi-source external knowledge from a legal knowledge base, a conversational LLM, and legal articles. Our method achieves state-of-the-art results on CAIL-2018, the largest legal charge prediction dataset.
  arXiv  Detail & Related papers  (2024-08-05T04:53:17Z)
• Explainable machine learning multi-label classification of Spanish legal judgements [6.817247544942709]
  We propose a hybrid system that applies Machine Learning for multi-label classification of judgements (sentences) and visual and natural language descriptions for explanation purposes. Our solution achieves over 85 % micro precision on a labelled data set annotated by legal experts.
  arXiv  Detail & Related papers  (2024-05-27T19:16:42Z)
• Empowering Prior to Court Legal Analysis: A Transparent and Accessible Dataset for Defensive Statement Classification and Interpretation [5.646219481667151]
  This paper introduces a novel dataset tailored for classification of statements made during police interviews, prior to court proceedings. We introduce a fine-tuned DistilBERT model that achieves state-of-the-art performance in distinguishing truthful from deceptive statements. We also present an XAI interface that empowers both legal professionals and non-specialists to interact with and benefit from our system.
  arXiv  Detail & Related papers  (2024-05-17T11:22:27Z)
• AutoGuide: Automated Generation and Selection of Context-Aware Guidelines for Large Language Model Agents [74.17623527375241]
  We introduce a novel framework, called AutoGuide, which automatically generates context-aware guidelines from offline experiences. As a result, our guidelines facilitate the provision of relevant knowledge for the agent's current decision-making process. Our evaluation demonstrates that AutoGuide significantly outperforms competitive baselines in complex benchmark domains.
  arXiv  Detail & Related papers  (2024-03-13T22:06:03Z)
• LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset [0.0]
  This study addresses the gap in the literature working with large legal corpora about how to isolate cases, in our case summary judgments, from a large corpus of UK court decisions. We use the Cambridge Law Corpus of 356,011 UK court decisions and determine that the large language model achieves a weighted F1 score of 0.94 versus 0.78 for keywords. We identify and extract 3,102 summary judgment cases, enabling us to map their distribution across various UK courts over a temporal span.
  arXiv  Detail & Related papers  (2024-03-04T10:13:30Z)
• Perspectives on Large Language Models for Relevance Judgment [56.935731584323996]
  Large language models (LLMs) claim that they can assist with relevance judgments. It is not clear whether automated judgments can reliably be used in evaluations of retrieval systems.
  arXiv  Detail & Related papers  (2023-04-13T13:08:38Z)
• AUTOLEX: An Automatic Framework for Linguistic Exploration [93.89709486642666]
  We propose an automatic framework that aims to ease linguists' discovery and extraction of concise descriptions of linguistic phenomena. Specifically, we apply this framework to extract descriptions for three phenomena: morphological agreement, case marking, and word order. We evaluate the descriptions with the help of language experts and propose a method for automated evaluation when human evaluation is infeasible.
  arXiv  Detail & Related papers  (2022-03-25T20:37:30Z)
• Curious Case of Language Generation Evaluation Metrics: A Cautionary Tale [52.663117551150954]
  A few popular metrics remain as the de facto metrics to evaluate tasks such as image captioning and machine translation. This is partly due to ease of use, and partly because researchers expect to see them and know how to interpret them. In this paper, we urge the community for more careful consideration of how they automatically evaluate their models.
  arXiv  Detail & Related papers  (2020-10-26T13:57:20Z)
• How Context Affects Language Models' Factual Predictions [134.29166998377187]
  We integrate information from a retrieval system with a pre-trained language model in a purely unsupervised way. We report that augmenting pre-trained language models in this way dramatically improves performance and that the resulting system, despite being unsupervised, is competitive with a supervised machine reading baseline.
  arXiv  Detail & Related papers  (2020-05-10T09:28:12Z)
• Interpretability Analysis for Named Entity Recognition to Understand System Predictions and How They Can Improve [49.878051587667244]
  We examine the performance of several variants of LSTM-CRF architectures for named entity recognition. We find that context representations do contribute to system performance, but that the main factor driving high performance is learning the name tokens themselves. We enlist human annotators to evaluate the feasibility of inferring entity types from the context alone and find that, while people are not able to infer the entity type either for the majority of the errors made by the context-only system, there is some room for improvement.
  arXiv  Detail & Related papers  (2020-04-09T14:37:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.