Parameter-Efficient Legal Domain Adaptation
- URL: http://arxiv.org/abs/2210.13712v1
- Date: Tue, 25 Oct 2022 02:14:15 GMT
- Title: Parameter-Efficient Legal Domain Adaptation
- Authors: Jonathan Li, Rohan Bhambhoria, Xiaodan Zhu
- Abstract summary: We propose a parameter-efficient legal domain adaptation, which uses vast unsupervised legal data from public legal forums to perform legal pre-training.
Our method exceeds or matches the fewshot performance of existing models while tuning only approximately 0.1% of model parameters.
To the best of our knowledge, this work is among the first to explore parameter-efficient methods of tuning language models toward the legal domain.
- Score: 39.51442413250532
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Seeking legal advice is often expensive. Recent advancement in machine
learning for solving complex problems can be leveraged to help make legal
services more accessible to the public. However, real-life applications
encounter significant challenges. State-of-the-art language models are growing
increasingly large, making parameter-efficient learning increasingly important.
Unfortunately, parameter-efficient methods perform poorly with small amounts of
data, which are common in the legal domain (where data labelling costs are
high). To address these challenges, we propose parameter-efficient legal domain
adaptation, which uses vast unsupervised legal data from public legal forums to
perform legal pre-training. This method exceeds or matches the fewshot
performance of existing models such as LEGAL-BERT on various legal tasks while
tuning only approximately 0.1% of model parameters. Additionally, we show that
our method can achieve calibration comparable to existing methods across
several tasks. To the best of our knowledge, this work is among the first to
explore parameter-efficient methods of tuning language models toward the legal
domain.
Related papers
- InternLM-Law: An Open Source Chinese Legal Large Language Model [72.2589401309848]
InternLM-Law is a specialized LLM tailored for addressing diverse legal queries related to Chinese laws.
We meticulously construct a dataset in the Chinese legal domain, encompassing over 1 million queries.
InternLM-Law achieves the highest average performance on LawBench, outperforming state-of-the-art models, including GPT-4, on 13 out of 20 subtasks.
arXiv Detail & Related papers (2024-06-21T06:19:03Z) - Query-driven Relevant Paragraph Extraction from Legal Judgments [1.2562034805037443]
Legal professionals often grapple with navigating lengthy legal judgements to pinpoint information that directly address their queries.
This paper focus on this task of extracting relevant paragraphs from legal judgements based on the query.
We construct a specialized dataset for this task from the European Court of Human Rights (ECtHR) using the case law guides.
arXiv Detail & Related papers (2024-03-31T08:03:39Z) - Automated Argument Generation from Legal Facts [6.057773749499076]
The number of cases submitted to the law system is far greater than the available number of legal professionals in a country.
In this study we partcularly focus on helping legal professionals in the process of analyzing a legal case.
Experimental results show that the generated arguments from the best performing method have on average 63% overlap with the benchmark set gold standard annotations.
arXiv Detail & Related papers (2023-10-09T12:49:35Z) - Interpretable Long-Form Legal Question Answering with
Retrieval-Augmented Large Language Models [10.834755282333589]
Long-form Legal Question Answering dataset comprises 1,868 expert-annotated legal questions in the French language.
Our experimental results demonstrate promising performance on automatic evaluation metrics.
As one of the only comprehensive, expert-annotated long-form LQA dataset, LLeQA has the potential to not only accelerate research towards resolving a significant real-world issue, but also act as a rigorous benchmark for evaluating NLP models in specialized domains.
arXiv Detail & Related papers (2023-09-29T08:23:19Z) - Chatlaw: A Multi-Agent Collaborative Legal Assistant with Knowledge Graph Enhanced Mixture-of-Experts Large Language Model [30.30848216845138]
Chatlaw is an innovative legal assistant utilizing a Mixture-of-Experts (MoE) model and a multi-agent system.
By integrating knowledge graphs with artificial screening, we construct a high-quality legal dataset to train the MoE model.
Our MoE model outperforms GPT-4 in the Lawbench and Unified Exam Qualification for Legal Professionals by 7.73% in accuracy and 11 points, respectively.
arXiv Detail & Related papers (2023-06-28T10:48:34Z) - SAILER: Structure-aware Pre-trained Language Model for Legal Case
Retrieval [75.05173891207214]
Legal case retrieval plays a core role in the intelligent legal system.
Most existing language models have difficulty understanding the long-distance dependencies between different structures.
We propose a new Structure-Aware pre-traIned language model for LEgal case Retrieval.
arXiv Detail & Related papers (2023-04-22T10:47:01Z) - Canary in a Coalmine: Better Membership Inference with Ensembled
Adversarial Queries [53.222218035435006]
We use adversarial tools to optimize for queries that are discriminative and diverse.
Our improvements achieve significantly more accurate membership inference than existing methods.
arXiv Detail & Related papers (2022-10-19T17:46:50Z) - Towards a Unified View of Parameter-Efficient Transfer Learning [108.94786930869473]
Fine-tuning large pre-trained language models on downstream tasks has become the de-facto learning paradigm in NLP.
Recent work has proposed a variety of parameter-efficient transfer learning methods that only fine-tune a small number of (extra) parameters to attain strong performance.
We break down the design of state-of-the-art parameter-efficient transfer learning methods and present a unified framework that establishes connections between them.
arXiv Detail & Related papers (2021-10-08T20:22:26Z) - LegaLMFiT: Efficient Short Legal Text Classification with LSTM Language
Model Pre-Training [0.0]
Large Transformer-based language models such as BERT have led to broad performance improvements on many NLP tasks.
In legal NLP, BERT-based models have led to new state-of-the-art results on multiple tasks.
We show that lightweight LSTM-based Language Models are able to capture enough information from a small legal text pretraining corpus and achieve excellent performance on short legal text classification tasks.
arXiv Detail & Related papers (2021-09-02T14:45:04Z) - Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents [56.40163943394202]
We release the Longformer-based pre-trained language model, named as Lawformer, for Chinese legal long documents understanding.
We evaluate Lawformer on a variety of LegalAI tasks, including judgment prediction, similar case retrieval, legal reading comprehension, and legal question answering.
arXiv Detail & Related papers (2021-05-09T09:39:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.