Pre-training, Fine-tuning and Re-ranking: A Three-Stage Framework for Legal Question Answering

• URL: http://arxiv.org/abs/2412.19482v1
• Date: Fri, 27 Dec 2024 06:33:42 GMT
• Title: Pre-training, Fine-tuning and Re-ranking: A Three-Stage Framework for Legal Question Answering
• Authors: Shiwen Ni, Hao Cheng, Min Yang,
• Abstract summary: Legal question answering (QA) has attracted increasing attention from people seeking legal advice. Previous methods mainly use a dual-encoder architecture to learn dense representations of both questions and answers. We propose a three-stage framework for underlinepre-training, underlinefine-tuning and underlinere-ranking.
• Score: 20.948737566388036
• License:
• Abstract: Legal question answering (QA) has attracted increasing attention from people seeking legal advice, which aims to retrieve the most applicable answers from a large-scale database of question-answer pairs. Previous methods mainly use a dual-encoder architecture to learn dense representations of both questions and answers. However, these methods could suffer from lacking domain knowledge and sufficient labeled training data. In this paper, we propose a three-stage (\underline{p}re-training, \underline{f}ine-tuning and \underline{r}e-ranking) framework for \underline{l}egal \underline{QA} (called PFR-LQA), which promotes the fine-grained text representation learning and boosts the performance of dense retrieval with the dual-encoder architecture. Concretely, we first conduct domain-specific pre-training on legal questions and answers through a self-supervised training objective, allowing the pre-trained model to be adapted to the legal domain. Then, we perform task-specific fine-tuning of the dual-encoder on legal question-answer pairs by using the supervised learning objective, leading to a high-quality dual-encoder for the specific downstream QA task. Finally, we employ a contextual re-ranking objective to further refine the output representations of questions produced by the document encoder, which uses contextual similarity to increase the discrepancy between the anchor and hard negative samples for better question re-ranking. We conduct extensive experiments on a manually annotated legal QA dataset. Experimental results show that our PFR-LQA method achieves better performance than the strong competitors for legal question answering.

Related papers

• FiTs: Fine-grained Two-stage Training for Knowledge-aware Question Answering [47.495991137191425]
  We propose a Fine-grained Two-stage training framework (FiTs) to boost the KAQA system performance. The first stage aims at aligning representations from the PLM and the KG, thus bridging the modality gaps between them. The second stage, called knowledge-aware fine-tuning, aims to improve the model's joint reasoning ability.
  arXiv  Detail & Related papers  (2023-02-23T06:25:51Z)
• Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization [89.04537372465612]
  Socratic pretraining is a question-driven, unsupervised pretraining objective designed to improve controllability in summarization tasks. Our results show that Socratic pretraining cuts task-specific labeled data requirements in half.
  arXiv  Detail & Related papers  (2022-12-20T17:27:10Z)
• UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph [89.98762327725112]
  Multi-hop Question Answering over Knowledge Graph(KGQA) aims to find the answer entities that are multiple hops away from the topic entities mentioned in a natural language question. We propose UniKGQA, a novel approach for multi-hop KGQA task, by unifying retrieval and reasoning in both model architecture and parameter learning.
  arXiv  Detail & Related papers  (2022-12-02T04:08:09Z)
• Miko Team: Deep Learning Approach for Legal Question Answering in ALQAC 2022 [2.242125769416219]
  We introduce efficient deep learning-based methods for legal document processing in the Automated Legal Question Answering Competition (ALQAC 2022) Our method is based on the XLM-RoBERTa model that is pre-trained from a large amount of unlabeled corpus before fine-tuning to the specific tasks. The experimental results showed that our method works well in legal retrieval information tasks with limited labeled data.
  arXiv  Detail & Related papers  (2022-11-04T00:50:20Z)
• Questions Are All You Need to Train a Dense Passage Retriever [123.13872383489172]
  ART is a new corpus-level autoencoding approach for training dense retrieval models that does not require any labeled training data. It uses a new document-retrieval autoencoding scheme, where (1) an input question is used to retrieve a set of evidence documents, and (2) the documents are then used to compute the probability of reconstructing the original question.
  arXiv  Detail & Related papers  (2022-06-21T18:16:31Z)
• Enhancing Dual-Encoders with Question and Answer Cross-Embeddings for Answer Retrieval [29.16807969384253]
  Dual-Encoders is a promising mechanism for answer retrieval in question answering (QA) systems. We propose a framework to enhance the Dual-Encoders model with question answer cross-embeddings and a novel Geometry Alignment Mechanism (GAM) Our framework significantly improves Dual-Encoders model and outperforms the state-of-the-art method on multiple answer retrieval datasets.
  arXiv  Detail & Related papers  (2022-06-07T02:39:24Z)
• Modeling Multi-hop Question Answering as Single Sequence Prediction [88.72621430714985]
  We propose a simple generative approach (PathFid) that extends the task beyond just answer generation. PathFid explicitly models the reasoning process to resolve the answer for multi-hop questions. Our experiments demonstrate that PathFid leads to strong performance gains on two multi-hop QA datasets.
  arXiv  Detail & Related papers  (2022-05-18T21:57:59Z)
• Joint Answering and Explanation for Visual Commonsense Reasoning [46.44588492897933]
  Visual Commonsense Reasoning endeavors to pursue a more high-level visual comprehension. It is composed of two indispensable processes: question answering over a given image and rationale inference for answer explanation. We present a plug-and-play knowledge distillation enhanced framework to couple the question answering and rationale inference processes.
  arXiv  Detail & Related papers  (2022-02-25T11:26:52Z)
• Weakly Supervised Pre-Training for Multi-Hop Retriever [23.79574380039197]
  We propose a new method for weakly supervised multi-hop retriever pre-training without human efforts. Our method includes 1) a pre-training task for generating vector representations of complex questions, 2) a scalable data generation method that produces the nested structure of question and sub-question as weak supervision for pre-training, and 3) a pre-training model structure based on dense encoders.
  arXiv  Detail & Related papers  (2021-06-18T08:06:02Z)
• Question Answering Infused Pre-training of General-Purpose Contextualized Representations [70.62967781515127]
  We propose a pre-training objective based on question answering (QA) for learning general-purpose contextual representations. We accomplish this goal by training a bi-encoder QA model, which independently encodes passages and questions, to match the predictions of a more accurate cross-encoder model. We show large improvements over both RoBERTa-large and previous state-of-the-art results on zero-shot and few-shot paraphrase detection.
  arXiv  Detail & Related papers  (2021-06-15T14:45:15Z)
• Unsupervised Alignment-based Iterative Evidence Retrieval for Multi-hop Question Answering [40.58976291178477]
  We introduce a simple, fast, and unsupervised iterative evidence retrieval method. Despite its simplicity, our approach outperforms all the previous methods on the evidence selection task. When these evidence sentences are fed into a RoBERTa answer classification component, we achieve state-of-the-art QA performance.
  arXiv  Detail & Related papers  (2020-05-04T00:19:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.