Customizing Contextualized Language Models forLegal Document Reviews
- URL: http://arxiv.org/abs/2102.05757v1
- Date: Wed, 10 Feb 2021 22:14:15 GMT
- Title: Customizing Contextualized Language Models forLegal Document Reviews
- Authors: Shohreh Shaghaghian, Luna (Yue) Feng, Borna Jafarpour, Nicolai
Pogrebnyakov
- Abstract summary: We show how different language models strained on general-domain corpora can be best customized for legal document reviewing tasks.
We compare their efficiencies with respect to task performances and present practical considerations.
- Score: 0.22940141855172028
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Inspired by the inductive transfer learning on computer vision, many efforts
have been made to train contextualized language models that boost the
performance of natural language processing tasks. These models are mostly
trained on large general-domain corpora such as news, books, or
Wikipedia.Although these pre-trained generic language models well perceive the
semantic and syntactic essence of a language structure, exploiting them in a
real-world domain-specific scenario still needs some practical considerations
to be taken into account such as token distribution shifts, inference time,
memory, and their simultaneous proficiency in multiple tasks. In this paper, we
focus on the legal domain and present how different language model strained on
general-domain corpora can be best customized for multiple legal document
reviewing tasks. We compare their efficiencies with respect to task
performances and present practical considerations.
Related papers
- Prompting Encoder Models for Zero-Shot Classification: A Cross-Domain Study in Italian [75.94354349994576]
This paper explores the feasibility of employing smaller, domain-specific encoder LMs alongside prompting techniques to enhance performance in specialized contexts.
Our study concentrates on the Italian bureaucratic and legal language, experimenting with both general-purpose and further pre-trained encoder-only models.
The results indicate that while further pre-trained models may show diminished robustness in general knowledge, they exhibit superior adaptability for domain-specific tasks, even in a zero-shot setting.
arXiv Detail & Related papers (2024-07-30T08:50:16Z) - Adapting Large Language Models to Domains via Reading Comprehension [86.24451681746676]
We explore how continued pre-training on domain-specific corpora influences large language models.
We show that training on the raw corpora endows the model with domain knowledge, but drastically hurts its ability for question answering.
We propose a simple method for transforming raw corpora into reading comprehension texts.
arXiv Detail & Related papers (2023-09-18T07:17:52Z) - On the cross-lingual transferability of multilingual prototypical models
across NLU tasks [2.44288434255221]
Supervised deep learning-based approaches have been applied to task-oriented dialog and have proven to be effective for limited domain and language applications.
In practice, these approaches suffer from the drawbacks of domain-driven design and under-resourced languages.
This article proposes to investigate the cross-lingual transferability of using synergistically few-shot learning with prototypical neural networks and multilingual Transformers-based models.
arXiv Detail & Related papers (2022-07-19T09:55:04Z) - Pre-Trained Language Models for Interactive Decision-Making [72.77825666035203]
We describe a framework for imitation learning in which goals and observations are represented as a sequence of embeddings.
We demonstrate that this framework enables effective generalization across different environments.
For test tasks involving novel goals or novel scenes, initializing policies with language models improves task completion rates by 43.6%.
arXiv Detail & Related papers (2022-02-03T18:55:52Z) - Analyzing the Limits of Self-Supervision in Handling Bias in Language [52.26068057260399]
We evaluate how well language models capture the semantics of four tasks for bias: diagnosis, identification, extraction and rephrasing.
Our analyses indicate that language models are capable of performing these tasks to widely varying degrees across different bias dimensions, such as gender and political affiliation.
arXiv Detail & Related papers (2021-12-16T05:36:08Z) - Lex Rosetta: Transfer of Predictive Models Across Languages,
Jurisdictions, and Legal Domains [40.58709137006848]
We analyze the use of Language-Agnostic Sentence Representations in sequence labeling models using Gated Recurrent Units (GRUs) that are transferable across languages.
We found that models generalize beyond the contexts on which they were trained.
We found that training the models on multiple contexts increases robustness and improves overall performance when evaluating on previously unseen contexts.
arXiv Detail & Related papers (2021-12-15T04:53:13Z) - Cross-Lingual Fine-Grained Entity Typing [26.973783464706447]
We present a unified cross-lingual fine-grained entity typing model capable of handling over 100 languages.
We analyze this model's ability to generalize to languages and entities unseen during training.
arXiv Detail & Related papers (2021-10-15T03:22:30Z) - Specializing Multilingual Language Models: An Empirical Study [50.7526245872855]
Contextualized word representations from pretrained multilingual language models have become the de facto standard for addressing natural language tasks.
For languages rarely or never seen by these models, directly using such models often results in suboptimal representation or use of data.
arXiv Detail & Related papers (2021-06-16T18:13:55Z) - Constrained Language Models Yield Few-Shot Semantic Parsers [73.50960967598654]
We explore the use of large pretrained language models as few-shot semantics.
The goal in semantic parsing is to generate a structured meaning representation given a natural language input.
We use language models to paraphrase inputs into a controlled sublanguage resembling English that can be automatically mapped to a target meaning representation.
arXiv Detail & Related papers (2021-04-18T08:13:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.