Comparing the Performance of NLP Toolkits and Evaluation measures in Legal Tech

• URL: http://arxiv.org/abs/2103.11792v1
• Date: Fri, 12 Mar 2021 11:06:32 GMT
• Title: Comparing the Performance of NLP Toolkits and Evaluation measures in Legal Tech
• Authors: Muhammad Zohaib Khan
• Abstract summary: We compare and analyze the pretrained Neural Language Models, XLNet (autoregressive), and BERT (autoencoder) on the Legal Tasks. XLNet Model performs better on our Sequence Classification task of Legal Opinions Classification, whereas BERT produces better results on the NER task. We use domain-specific pretraining and additional legal vocabulary to adapt BERT Model further to the Legal Domain.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent developments in Natural Language Processing have led to the introduction of state-of-the-art Neural Language Models, enabled with unsupervised transferable learning, using different pretraining objectives. While these models achieve excellent results on the downstream NLP tasks, various domain adaptation techniques can improve their performance on domain-specific tasks. We compare and analyze the pretrained Neural Language Models, XLNet (autoregressive), and BERT (autoencoder) on the Legal Tasks. Results show that XLNet Model performs better on our Sequence Classification task of Legal Opinions Classification, whereas BERT produces better results on the NER task. We use domain-specific pretraining and additional legal vocabulary to adapt BERT Model further to the Legal Domain. We prepared multiple variants of the BERT Model, using both methods and their combination. Comparing our variants of the BERT Model, specializing in the Legal Domain, we conclude that both additional pretraining and vocabulary techniques enhance the BERT model's performance on the Legal Opinions Classification task. Additional legal vocabulary improves BERT's performance on the NER task. Combining the pretraining and vocabulary techniques further improves the final results. Our Legal-Vocab-BERT Model gives the best results on the Legal Opinions Task, outperforming the larger pretrained general Language Models, i.e., BERT-Base and XLNet-Base.

Related papers

• LegalTurk Optimized BERT for Multi-Label Text Classification and NER [0.0]
  We introduce our innovative modified pre-training approach by combining diverse masking strategies. In this work, we focus on two essential downstream tasks in the legal domain: name entity recognition and multi-label text classification. Our modified approach demonstrated significant improvements in both NER and multi-label text classification tasks compared to the original BERT model.
  arXiv  Detail & Related papers  (2024-06-30T10:19:54Z)
• Test-Time Training on Graphs with Large Language Models (LLMs) [68.375487369596]
  Test-Time Training (TTT) has been proposed as a promising approach to train Graph Neural Networks (GNNs) Inspired by the great annotation ability of Large Language Models (LLMs) on Text-Attributed Graphs (TAGs), we propose to enhance the test-time training on graphs with LLMs as annotators. A two-stage training strategy is designed to tailor the test-time model with the limited and noisy labels.
  arXiv  Detail & Related papers  (2024-04-21T08:20:02Z)
• Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis [84.12658971655253]
  We propose Adapted Multimodal BERT, a BERT-based architecture for multimodal tasks. adapter adjusts the pretrained language model for the task at hand, while the fusion layers perform task-specific, layer-wise fusion of audio-visual information with textual BERT representations. In our ablations we see that this approach leads to efficient models, that can outperform their fine-tuned counterparts and are robust to input noise.
  arXiv  Detail & Related papers  (2022-12-01T17:31:42Z)
• AraLegal-BERT: A pretrained language model for Arabic Legal text [0.399013650624183]
  We introduce AraLegal-BERT, a bidirectional encoder Transformer-based model that have been thoroughly tested and carefully optimized. We fine-tuned AraLegal-BERT and evaluated it against three BERT variations for Arabic language in three natural languages understanding (NLU) tasks. The results show that the base version of AraLegal-BERT achieve better accuracy than the general and original BERT over the Legal text.
  arXiv  Detail & Related papers  (2022-10-15T13:08:40Z)
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization [94.4409074435894]
  We propose a novel and effective fine-tuning framework, named Layerwise Noise Stability Regularization (LNSR) Specifically, we propose to inject the standard Gaussian noise and regularize hidden representations of the fine-tuned model. We demonstrate the advantages of the proposed method over other state-of-the-art algorithms including L2-SP, Mixout and SMART.
  arXiv  Detail & Related papers  (2022-06-12T04:42:49Z)
• Legal Transformer Models May Not Always Help [3.6061626009104057]
  This work investigates the value of domain adaptive pre-training and language adapters in legal NLP tasks. We show that domain adaptive pre-training is only helpful with low-resource downstream tasks. As an additional result, we release LegalRoBERTa, a RoBERTa model further pre-trained on legal corpora.
  arXiv  Detail & Related papers  (2021-09-14T17:53:55Z)
• LegaLMFiT: Efficient Short Legal Text Classification with LSTM Language Model Pre-Training [0.0]
  Large Transformer-based language models such as BERT have led to broad performance improvements on many NLP tasks. In legal NLP, BERT-based models have led to new state-of-the-art results on multiple tasks. We show that lightweight LSTM-based Language Models are able to capture enough information from a small legal text pretraining corpus and achieve excellent performance on short legal text classification tasks.
  arXiv  Detail & Related papers  (2021-09-02T14:45:04Z)
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing [73.37262264915739]
  We show that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks.
  arXiv  Detail & Related papers  (2020-07-31T00:04:15Z)
• DagoBERT: Generating Derivational Morphology with a Pretrained Language Model [20.81930455526026]
  We show that pretrained language models (PLMs) can generate derivationally complex words. Our best model, DagoBERT, clearly outperforms the previous state of the art in derivation generation. Our experiments show that the input segmentation crucially impacts BERT's derivational knowledge.
  arXiv  Detail & Related papers  (2020-05-02T01:26:46Z)
• Coreferential Reasoning Learning for Language Representation [88.14248323659267]
  We present CorefBERT, a novel language representation model that can capture the coreferential relations in context. The experimental results show that, compared with existing baseline models, CorefBERT can achieve significant improvements consistently on various downstream NLP tasks.
  arXiv  Detail & Related papers  (2020-04-15T03:57:45Z)
• Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation [84.64004917951547]
  Fine-tuning pre-trained language models like BERT has become an effective way in NLP. In this paper, we improve the fine-tuning of BERT with two effective mechanisms: self-ensemble and self-distillation.
  arXiv  Detail & Related papers  (2020-02-24T16:17:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.