Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice

• URL: http://arxiv.org/abs/2409.07713v1
• Date: Thu, 12 Sep 2024 02:40:28 GMT
• Title: Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice
• Authors: Jonathan Li, Rohan Bhambhoria, Samuel Dahan, Xiaodan Zhu,
• Abstract summary: We propose a human-centric legal NLP pipeline, covering data sourcing, inference, and evaluation. We release a dataset, LegalQA, with real and specific legal questions spanning from employment law to criminal law. We show that retrieval-augmented generation from only 850 citations in the train set can match or outperform internet-wide retrieval.
• Score: 32.550204238857724
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative AI models, such as the GPT and Llama series, have significant potential to assist laypeople in answering legal questions. However, little prior work focuses on the data sourcing, inference, and evaluation of these models in the context of laypersons. To this end, we propose a human-centric legal NLP pipeline, covering data sourcing, inference, and evaluation. We introduce and release a dataset, LegalQA, with real and specific legal questions spanning from employment law to criminal law, corresponding answers written by legal experts, and citations for each answer. We develop an automatic evaluation protocol for this dataset, then show that retrieval-augmented generation from only 850 citations in the train set can match or outperform internet-wide retrieval, despite containing 9 orders of magnitude less data. Finally, we propose future directions for open-sourced efforts, which fall behind closed-sourced models.

Related papers

• Enhancing Legal Case Retrieval via Scaling High-quality Synthetic Query-Candidate Pairs [67.54302101989542]
  Legal case retrieval aims to provide similar cases as references for a given fact description. Existing works mainly focus on case-to-case retrieval using lengthy queries. Data scale is insufficient to satisfy the training requirements of existing data-hungry neural models.
  arXiv  Detail & Related papers  (2024-10-09T06:26:39Z)
• Evaluating Copyright Takedown Methods for Language Models [100.38129820325497]
  Language models (LMs) derive their capabilities from extensive training on diverse data, including potentially copyrighted material. This paper introduces the first evaluation of the feasibility and side effects of copyright takedowns for LMs. We examine several strategies, including adding system prompts, decoding-time filtering interventions, and unlearning approaches.
  arXiv  Detail & Related papers  (2024-06-26T18:09:46Z)
• STARD: A Chinese Statute Retrieval Dataset with Real Queries Issued by Non-professionals [14.002280587675175]
  Statute retrieval aims to find relevant statutory articles for specific queries. Existing statute retrieval benchmarks focus on formal and professional queries from sources like bar exams and legal case documents. To address this gap, we introduce the STAtute Retrieval dataset (STARD) Unlike existing statute retrieval datasets, STARD captures the complexity and diversity of real queries from the general public.
  arXiv  Detail & Related papers  (2024-06-21T17:10:09Z)
• Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models [10.834755282333589]
  Long-form Legal Question Answering dataset comprises 1,868 expert-annotated legal questions in the French language. Our experimental results demonstrate promising performance on automatic evaluation metrics. As one of the only comprehensive, expert-annotated long-form LQA dataset, LLeQA has the potential to not only accelerate research towards resolving a significant real-world issue, but also act as a rigorous benchmark for evaluating NLP models in specialized domains.
  arXiv  Detail & Related papers  (2023-09-29T08:23:19Z)
• Going beyond research datasets: Novel intent discovery in the industry setting [60.90117614762879]
  This paper proposes methods to improve the intent discovery pipeline deployed in a large e-commerce platform. We show the benefit of pre-training language models on in-domain data: both self-supervised and with weak supervision. We also devise the best method to utilize the conversational structure (i.e., question and answer) of real-life datasets during fine-tuning for clustering tasks, which we call Conv.
  arXiv  Detail & Related papers  (2023-05-09T14:21:29Z)
• The Legal Argument Reasoning Task in Civil Procedure [2.079168053329397]
  We present a new NLP task and dataset from the domain of the U.S. civil procedure. Each instance of the dataset consists of a general introduction to the case, a particular question, and a possible solution argument.
  arXiv  Detail & Related papers  (2022-11-05T17:41:00Z)
• Should I disclose my dataset? Caveats between reproducibility and individual data rights [5.816090284071069]
  Digital availability of court documents increases possibilities for researchers. However, personal data protection laws impose restrictions on data exposure. We present legal and ethical considerations on the issue, as well as guidelines for researchers.
  arXiv  Detail & Related papers  (2022-11-01T14:42:11Z)
• Algorithmic Fairness Datasets: the Story so Far [68.45921483094705]
  Data-driven algorithms are studied in diverse domains to support critical decisions, directly impacting people's well-being. A growing community of researchers has been investigating the equity of existing algorithms and proposing novel ones, advancing the understanding of risks and opportunities of automated decision-making for historically disadvantaged populations. Progress in fair Machine Learning hinges on data, which can be appropriately used only if adequately documented. Unfortunately, the algorithmic fairness community suffers from a collective data documentation debt caused by a lack of information on specific resources (opacity) and scatteredness of available information (sparsity)
  arXiv  Detail & Related papers  (2022-02-03T17:25:46Z)
• Yes-Yes-Yes: Donation-based Peer Reviewing Data Collection for ACL Rolling Review and Beyond [58.71736531356398]
  We present an in-depth discussion of peer reviewing data, outline the ethical and legal desiderata for peer reviewing data collection, and propose the first continuous, donation-based data collection workflow. We report on the ongoing implementation of this workflow at the ACL Rolling Review and deliver the first insights obtained with the newly collected data.
  arXiv  Detail & Related papers  (2022-01-27T11:02:43Z)
• Text-guided Legal Knowledge Graph Reasoning [11.089663225933412]
  We propose a novel legal application of legal provision prediction (LPP), which aims to predict the related legal provisions of affairs. We collect amounts of real-world legal provision data from the Guangdong government service website and construct a legal dataset called LegalLPP.
  arXiv  Detail & Related papers  (2021-04-06T04:42:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.