LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
- URL: http://arxiv.org/abs/2110.00976v1
- Date: Sun, 3 Oct 2021 10:50:51 GMT
- Title: LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
- Authors: Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion
Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras
- Abstract summary: We introduce the Legal General Language Evaluation (LexGLUE) benchmark, a collection of datasets for evaluating model performance across a diverse set of legal NLU tasks.
We also provide an evaluation and analysis of several generic and legal-oriented models demonstrating that the latter consistently offer performance improvements across multiple tasks.
- Score: 15.026117429782996
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Law, interpretations of law, legal arguments, agreements, etc. are typically
expressed in writing, leading to the production of vast corpora of legal text.
Their analysis, which is at the center of legal practice, becomes increasingly
elaborate as these collections grow in size. Natural language understanding
(NLU) technologies can be a valuable tool to support legal practitioners in
these endeavors. Their usefulness, however, largely depends on whether current
state-of-the-art models can generalize across various tasks in the legal
domain. To answer this currently open question, we introduce the Legal General
Language Understanding Evaluation (LexGLUE) benchmark, a collection of datasets
for evaluating model performance across a diverse set of legal NLU tasks in a
standardized way. We also provide an evaluation and analysis of several generic
and legal-oriented models demonstrating that the latter consistently offer
performance improvements across multiple tasks.
Related papers
- InternLM-Law: An Open Source Chinese Legal Large Language Model [72.2589401309848]
InternLM-Law is a specialized LLM tailored for addressing diverse legal queries related to Chinese laws.
We meticulously construct a dataset in the Chinese legal domain, encompassing over 1 million queries.
InternLM-Law achieves the highest average performance on LawBench, outperforming state-of-the-art models, including GPT-4, on 13 out of 20 subtasks.
arXiv Detail & Related papers (2024-06-21T06:19:03Z) - BLT: Can Large Language Models Handle Basic Legal Text? [50.46167465931653]
GPT-4, Claude, and PaLM 2 perform poorly at basic legal text handling.
Fine-tuning for these tasks brings even a smaller model to near-perfect performance on our test set.
arXiv Detail & Related papers (2023-11-16T09:09:22Z) - Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model
Collaboration [52.57055162778548]
Legal Judgment Prediction (LJP) has become an increasingly crucial task in Legal AI.
Precedents are the previous legal cases with similar facts, which are the basis for the judgment of the subsequent case in national legal systems.
Recent advances in deep learning have enabled a variety of techniques to be used to solve the LJP task.
arXiv Detail & Related papers (2023-10-13T16:47:20Z) - SCALE: Scaling up the Complexity for Advanced Language Model Evaluation [19.339580164451256]
We introduce a novel NLP benchmark that poses challenges to current Large Language Models (LLMs)
Our benchmark comprises diverse legal NLP datasets from the Swiss legal system.
As part of our study, we evaluate several pre-trained multilingual language models on our benchmark to establish strong baselines as a point of reference.
arXiv Detail & Related papers (2023-06-15T16:19:15Z) - LeXFiles and LegalLAMA: Facilitating English Multinational Legal
Language Model Development [8.931169262582442]
We conduct a detailed analysis on the performance of legal-oriented pre-trained language models (PLMs)
We examine the interplay between their original objective, acquired knowledge, and legal language understanding capacities.
We find that probing performance strongly correlates with upstream performance in related legal topics.
arXiv Detail & Related papers (2023-05-12T14:21:38Z) - Unlocking Practical Applications in Legal Domain: Evaluation of GPT for
Zero-Shot Semantic Annotation of Legal Texts [0.0]
We evaluate the capability of a state-of-the-art generative pre-trained transformer (GPT) model to perform semantic annotation of short text snippets.
We found that the GPT model performs surprisingly well in zero-shot settings on diverse types of documents.
arXiv Detail & Related papers (2023-05-08T01:55:53Z) - SAILER: Structure-aware Pre-trained Language Model for Legal Case
Retrieval [75.05173891207214]
Legal case retrieval plays a core role in the intelligent legal system.
Most existing language models have difficulty understanding the long-distance dependencies between different structures.
We propose a new Structure-Aware pre-traIned language model for LEgal case Retrieval.
arXiv Detail & Related papers (2023-04-22T10:47:01Z) - The Legal Argument Reasoning Task in Civil Procedure [2.079168053329397]
We present a new NLP task and dataset from the domain of the U.S. civil procedure.
Each instance of the dataset consists of a general introduction to the case, a particular question, and a possible solution argument.
arXiv Detail & Related papers (2022-11-05T17:41:00Z) - Benchmarking Generalization via In-Context Instructions on 1,600+
Language Tasks [95.06087720086133]
Natural-Instructions v2 is a collection of 1,600+ diverse language tasks and their expert written instructions.
The benchmark covers 70+ distinct task types, such as tagging, in-filling, and rewriting.
This benchmark enables large-scale evaluation of cross-task generalization of the models.
arXiv Detail & Related papers (2022-04-16T03:12:30Z) - Legal Search in Case Law and Statute Law [12.697393184074457]
We describe a method to identify document pairwise relevance in the context of a typical legal document collection.
We review the usage of generalized language models, including supervised and unsupervised learning.
arXiv Detail & Related papers (2021-08-23T12:51:24Z) - Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents [56.40163943394202]
We release the Longformer-based pre-trained language model, named as Lawformer, for Chinese legal long documents understanding.
We evaluate Lawformer on a variety of LegalAI tasks, including judgment prediction, similar case retrieval, legal reading comprehension, and legal question answering.
arXiv Detail & Related papers (2021-05-09T09:39:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.