Impacts of Continued Legal Pre-Training and IFT on LLMs' Latent Representations of Human-Defined Legal Concepts
- URL: http://arxiv.org/abs/2410.12001v1
- Date: Tue, 15 Oct 2024 19:06:14 GMT
- Title: Impacts of Continued Legal Pre-Training and IFT on LLMs' Latent Representations of Human-Defined Legal Concepts
- Authors: Shaun Ho,
- Abstract summary: We examined 7 distinct text sequences from recent AI & Law, each containing a human-defined legal concept.
We then visualized patterns of raw attention score alterations, evaluating whether legal training introduced novel attention patterns corresponding to structures of human legal knowledge.
This inquiry revealed that (1) the impact of legal training was unevenly distributed across the various human-defined legal concepts, and (2) the contextual representations of legal knowledge learned during legal training did not coincide with structures of human-defined legal concepts.
- Score: 0.0
- License:
- Abstract: This paper aims to offer AI & Law researchers and practitioners a more detailed understanding of whether and how continued pre-training and instruction fine-tuning (IFT) of large language models (LLMs) on legal corpora increases their utilization of human-defined legal concepts when developing global contextual representations of input sequences. We compared three models: Mistral 7B, SaulLM-7B-Base (Mistral 7B with continued pre-training on legal corpora), and SaulLM-7B-Instruct (with further IFT). This preliminary assessment examined 7 distinct text sequences from recent AI & Law literature, each containing a human-defined legal concept. We first compared the proportions of total attention the models allocated to subsets of tokens representing the legal concepts. We then visualized patterns of raw attention score alterations, evaluating whether legal training introduced novel attention patterns corresponding to structures of human legal knowledge. This inquiry revealed that (1) the impact of legal training was unevenly distributed across the various human-defined legal concepts, and (2) the contextual representations of legal knowledge learned during legal training did not coincide with structures of human-defined legal concepts. We conclude with suggestions for further investigation into the dynamics of legal LLM training.
Related papers
- Automating Legal Concept Interpretation with LLMs: Retrieval, Generation, and Evaluation [27.345475442620746]
Legal articles often include vague concepts for adapting to the ever-changing society.
It requires meticulous and professional annotations and summarizations by legal experts.
By emulating legal experts' doctrinal method, we introduce a novel framework, ATRIE.
ATRIE comprises a legal concept interpreter and a legal concept interpretation evaluator.
arXiv Detail & Related papers (2025-01-03T10:11:38Z) - InternLM-Law: An Open Source Chinese Legal Large Language Model [72.2589401309848]
InternLM-Law is a specialized LLM tailored for addressing diverse legal queries related to Chinese laws.
We meticulously construct a dataset in the Chinese legal domain, encompassing over 1 million queries.
InternLM-Law achieves the highest average performance on LawBench, outperforming state-of-the-art models, including GPT-4, on 13 out of 20 subtasks.
arXiv Detail & Related papers (2024-06-21T06:19:03Z) - DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment [55.91429725404988]
We introduce DELTA, a discriminative model designed for legal case retrieval.
We leverage shallow decoders to create information bottlenecks, aiming to enhance the representation ability.
Our approach can outperform existing state-of-the-art methods in legal case retrieval.
arXiv Detail & Related papers (2024-03-27T10:40:14Z) - Leveraging Large Language Models for Learning Complex Legal Concepts through Storytelling [43.243889347008455]
We present a novel application of large language models (LLMs) in legal education to help non-experts learn intricate legal concepts through storytelling.
We introduce a new dataset LegalStories, which consists of 294 complex legal doctrines, each accompanied by a story and a set of multiple-choice questions.
We find that LLM-generated stories enhance comprehension of legal concepts and interest in law among non-native speakers compared to only definitions.
arXiv Detail & Related papers (2024-02-26T20:56:06Z) - Report of the 1st Workshop on Generative AI and Law [78.62063815165968]
This report presents the takeaways of the inaugural Workshop on Generative AI and Law (GenLaw)
A cross-disciplinary group of practitioners and scholars from computer science and law convened to discuss the technical, doctrinal, and policy challenges presented by law for Generative AI.
arXiv Detail & Related papers (2023-11-11T04:13:37Z) - Prototype-Based Interpretability for Legal Citation Prediction [16.660004925391842]
We design the task with parallels to the thought-process of lawyers, i.e., with reference to both precedents and legislative provisions.
After initial experimental results, we refine the target citation predictions with the feedback of legal experts.
We introduce a prototype architecture to add interpretability, achieving strong performance while adhering to decision parameters used by lawyers.
arXiv Detail & Related papers (2023-05-25T21:40:58Z) - SAILER: Structure-aware Pre-trained Language Model for Legal Case
Retrieval [75.05173891207214]
Legal case retrieval plays a core role in the intelligent legal system.
Most existing language models have difficulty understanding the long-distance dependencies between different structures.
We propose a new Structure-Aware pre-traIned language model for LEgal case Retrieval.
arXiv Detail & Related papers (2023-04-22T10:47:01Z) - Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents [56.40163943394202]
We release the Longformer-based pre-trained language model, named as Lawformer, for Chinese legal long documents understanding.
We evaluate Lawformer on a variety of LegalAI tasks, including judgment prediction, similar case retrieval, legal reading comprehension, and legal question answering.
arXiv Detail & Related papers (2021-05-09T09:39:25Z) - Legal Sentiment Analysis and Opinion Mining (LSAOM): Assimilating
Advances in Autonomous AI Legal Reasoning [0.0]
Legal Sentiment Analysis and Opinion Mining (LSAOM) consists of two often intertwined phenomena and actions underlying legal discussions and narratives.
Efforts to undertake LSAOM have historically been performed by human hand and cognition.
Advances in Artificial Intelligence (AI) involving especially Natural Language Processing (NLP) and Machine Learning (ML) are bolstering how automation can systematically perform either or both of Sentiment Analysis and Opinion Mining.
arXiv Detail & Related papers (2020-10-02T04:15:21Z) - Authorized and Unauthorized Practices of Law: The Role of Autonomous
Levels of AI Legal Reasoning [0.0]
The legal field has sought to define Authorized Practices of Law (APL) versus Unauthorized Practices of Law (UPL)
This paper explores a newly derived instrumental grid depicting the key characteristics underlying APL and UPL as they apply to the AILR autonomous levels.
arXiv Detail & Related papers (2020-08-19T18:35:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.