LEGAL-BERT: The Muppets straight out of Law School

• URL: http://arxiv.org/abs/2010.02559v1
• Date: Tue, 6 Oct 2020 09:06:07 GMT
• Title: LEGAL-BERT: The Muppets straight out of Law School
• Authors: Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras and Ion Androutsopoulos
• Abstract summary: We explore approaches for applying BERT models to downstream legal tasks, evaluating on multiple datasets. Our findings indicate that the previous guidelines for pre-training and fine-tuning, often blindly followed, do not always generalize well in the legal domain. We release LEGAL-BERT, a family of BERT models intended to assist legal NLP research, computational law, and legal technology applications.
• Score: 52.53830441117363
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: BERT has achieved impressive performance in several NLP tasks. However, there has been limited investigation on its adaptation guidelines in specialised domains. Here we focus on the legal domain, where we explore several approaches for applying BERT models to downstream legal tasks, evaluating on multiple datasets. Our findings indicate that the previous guidelines for pre-training and fine-tuning, often blindly followed, do not always generalize well in the legal domain. Thus we propose a systematic investigation of the available strategies when applying BERT in specialised domains. These are: (a) use the original BERT out of the box, (b) adapt BERT by additional pre-training on domain-specific corpora, and (c) pre-train BERT from scratch on domain-specific corpora. We also propose a broader hyper-parameter search space when fine-tuning for downstream tasks and we release LEGAL-BERT, a family of BERT models intended to assist legal NLP research, computational law, and legal technology applications.

Related papers

• InternLM-Law: An Open Source Chinese Legal Large Language Model [72.2589401309848]
  InternLM-Law is a specialized LLM tailored for addressing diverse legal queries related to Chinese laws. We meticulously construct a dataset in the Chinese legal domain, encompassing over 1 million queries. InternLM-Law achieves the highest average performance on LawBench, outperforming state-of-the-art models, including GPT-4, on 13 out of 20 subtasks.
  arXiv  Detail & Related papers  (2024-06-21T06:19:03Z)
• D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models [53.622682408251755]
  We propose the Scaling Law of the Domain-specific Continual Pre-Training (D-CPT Law) to decide the optimal mixture ratio with acceptable training costs. Specifically, by fitting the D-CPT Law, we can easily predict the general and downstream performance of arbitrary mixture ratios. We also extend our standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law.
  arXiv  Detail & Related papers  (2024-06-03T14:40:31Z)
• AraLegal-BERT: A pretrained language model for Arabic Legal text [0.399013650624183]
  We introduce AraLegal-BERT, a bidirectional encoder Transformer-based model that have been thoroughly tested and carefully optimized. We fine-tuned AraLegal-BERT and evaluated it against three BERT variations for Arabic language in three natural languages understanding (NLU) tasks. The results show that the base version of AraLegal-BERT achieve better accuracy than the general and original BERT over the Legal text.
  arXiv  Detail & Related papers  (2022-10-15T13:08:40Z)
• Enhancing Legal Argument Mining with Domain Pre-training and Neural Networks [0.45119235878273]
  The contextual word embedding model, BERT, has proved its ability on downstream tasks with limited quantities of annotated data. BERT and its variants help to reduce the burden of complex annotation work in many interdisciplinary research areas.
  arXiv  Detail & Related papers  (2022-02-27T21:24:53Z)
• Unsupervised Domain Generalization for Person Re-identification: A Domain-specific Adaptive Framework [50.88463458896428]
  Domain generalization (DG) has attracted much attention in person re-identification (ReID) recently. Existing methods usually need the source domains to be labeled, which could be a significant burden for practical ReID tasks. We propose a simple and efficient domain-specific adaptive framework, and realize it with an adaptive normalization module.
  arXiv  Detail & Related papers  (2021-11-30T02:35:51Z)
• JuriBERT: A Masked-Language Model Adaptation for French Legal Text [14.330469316695853]
  We focus on creating a language model adapted to French legal text with the goal of helping law professionals. We conclude that some specific tasks do not benefit from generic language models pre-trained on large amounts of data. We release JuriBERT, a new set of BERT models adapted to the French legal domain.
  arXiv  Detail & Related papers  (2021-10-04T14:51:24Z)
• When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset [2.0924876102146714]
  We present a new dataset comprised of over 53,000+ multiple choice questions to identify the relevant holding of a cited case. We show that domain pretraining may be warranted when the task exhibits sufficient similarity to the pretraining corpus. Our findings inform when researchers should engage resource-intensive pretraining and show that Transformer-based architectures, too, learn embeddings suggestive of distinct legal language.
  arXiv  Detail & Related papers  (2021-04-18T00:57:16Z)
• Comparing the Performance of NLP Toolkits and Evaluation measures in Legal Tech [0.0]
  We compare and analyze the pretrained Neural Language Models, XLNet (autoregressive), and BERT (autoencoder) on the Legal Tasks. XLNet Model performs better on our Sequence Classification task of Legal Opinions Classification, whereas BERT produces better results on the NER task. We use domain-specific pretraining and additional legal vocabulary to adapt BERT Model further to the Legal Domain.
  arXiv  Detail & Related papers  (2021-03-12T11:06:32Z)
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing [73.37262264915739]
  We show that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks.
  arXiv  Detail & Related papers  (2020-07-31T00:04:15Z)
• How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence [81.04070052740596]
  Legal Artificial Intelligence (LegalAI) focuses on applying the technology of artificial intelligence, especially natural language processing, to benefit tasks in the legal domain. This paper introduces the history, the current state, and the future directions of research in LegalAI.
  arXiv  Detail & Related papers  (2020-04-25T14:45:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.