Answering Questions in Stages: Prompt Chaining for Contract QA

• URL: http://arxiv.org/abs/2410.12840v1
• Date: Wed, 09 Oct 2024 17:14:13 GMT
• Title: Answering Questions in Stages: Prompt Chaining for Contract QA
• Authors: Adam Roegiest, Radha Chitta,
• Abstract summary: We propose two-stage prompt chaining to produce structured answers to multiple-choice and multiple-select questions. We analyze situations where this technique works well and areas where further refinement is needed.
• Score: 1.0359008237358598
• License:
• Abstract: Finding answers to legal questions about clauses in contracts is an important form of analysis in many legal workflows (e.g., understanding market trends, due diligence, risk mitigation) but more important is being able to do this at scale. Prior work showed that it is possible to use large language models with simple zero-shot prompts to generate structured answers to questions, which can later be incorporated into legal workflows. Such prompts, while effective on simple and straightforward clauses, fail to perform when the clauses are long and contain information not relevant to the question. In this paper, we propose two-stage prompt chaining to produce structured answers to multiple-choice and multiple-select questions and show that they are more effective than simple prompts on more nuanced legal text. We analyze situations where this technique works well and areas where further refinement is needed, especially when the underlying linguistic variations are more than can be captured by simply specifying possible answers. Finally, we discuss future research that seeks to refine this work by improving stage one results by making them more question-specific.

Related papers

• Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search [89.1772985740272]
  In mixed-initiative conversational search systems, clarifying questions are used to help users who struggle to express their intentions in a single query. We hypothesize that in scenarios where multimodal information is pertinent, the clarification process can be improved by using non-textual information. We collect a dataset named Melon that contains over 4k multimodal clarifying questions, enriched with over 14k images. Several analyses are conducted to understand the importance of multimodal contents during the query clarification phase.
  arXiv  Detail & Related papers  (2024-02-12T16:04:01Z)
• Estimating the Usefulness of Clarifying Questions and Answers for Conversational Search [17.0363715044341]
  We propose a method for processing answers to clarifying questions, moving away from previous work that simply appends answers to the original query. Specifically, we propose a classifier for assessing usefulness of the prompted clarifying question and an answer given by the user. Results demonstrate significant improvements over strong non-mixed-initiative baselines.
  arXiv  Detail & Related papers  (2024-01-21T11:04:30Z)
• A Search for Prompts: Generating Structured Answers from Contracts [40.99057706243682]
  We present a form of legal question answering that seeks to return one (or more) fixed answers for a question about a contract clause. We discuss our exploration methodology for legal question answering prompts using OpenAI's textitGPT-3.5-Turbo and provide a summary of insights.
  arXiv  Detail & Related papers  (2023-10-16T07:29:38Z)
• Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models [10.834755282333589]
  Long-form Legal Question Answering dataset comprises 1,868 expert-annotated legal questions in the French language. Our experimental results demonstrate promising performance on automatic evaluation metrics. As one of the only comprehensive, expert-annotated long-form LQA dataset, LLeQA has the potential to not only accelerate research towards resolving a significant real-world issue, but also act as a rigorous benchmark for evaluating NLP models in specialized domains.
  arXiv  Detail & Related papers  (2023-09-29T08:23:19Z)
• Elaborative Simplification as Implicit Questions Under Discussion [51.17933943734872]
  This paper proposes to view elaborative simplification through the lens of the Question Under Discussion (QUD) framework. We show that explicitly modeling QUD provides essential understanding of elaborative simplification and how the elaborations connect with the rest of the discourse.
  arXiv  Detail & Related papers  (2023-05-17T17:26:16Z)
• HPE:Answering Complex Questions over Text by Hybrid Question Parsing and Execution [92.69684305578957]
  We propose a framework of question parsing and execution on textual QA. The proposed framework can be viewed as a top-down question parsing followed by a bottom-up answer backtracking. Our experiments on MuSiQue, 2WikiQA, HotpotQA, and NQ show that the proposed parsing and hybrid execution framework outperforms existing approaches in supervised, few-shot, and zero-shot settings.
  arXiv  Detail & Related papers  (2023-05-12T22:37:06Z)
• Keeping the Questions Conversational: Using Structured Representations to Resolve Dependency in Conversational Question Answering [26.997542897342164]
  We propose a novel framework, CONVSR (CONVQA using Structured Representations) for capturing and generating intermediate representations as conversational cues. We test our model on the QuAC and CANARD datasets and illustrate by experimental results that our proposed framework achieves a better F1 score than the standard question rewriting model.
  arXiv  Detail & Related papers  (2023-04-14T13:42:32Z)
• Successive Prompting for Decomposing Complex Questions [50.00659445976735]
  Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting. We introduce Successive Prompting'', where we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution. Our best model (with successive prompting) achieves an improvement of 5% absolute F1 on a few-shot version of the DROP dataset.
  arXiv  Detail & Related papers  (2022-12-08T06:03:38Z)
• Asking It All: Generating Contextualized Questions for any Semantic Role [56.724302729493594]
  We introduce the task of role question generation, which is given a predicate mention and a passage. We develop a two-stage model for this task, which first produces a context-independent question prototype for each role. Our evaluation demonstrates that we generate diverse and well-formed questions for a large, broad-coverage of predicates and roles.
  arXiv  Detail & Related papers  (2021-09-10T12:31:14Z)
• Open-Retrieval Conversational Machine Reading [80.13988353794586]
  In conversational machine reading, systems need to interpret natural language rules, answer high-level questions, and ask follow-up clarification questions. Existing works assume the rule text is provided for each user question, which neglects the essential retrieval step in real scenarios. In this work, we propose and investigate an open-retrieval setting of conversational machine reading.
  arXiv  Detail & Related papers  (2021-02-17T08:55:01Z)
• Asking Complex Questions with Multi-hop Answer-focused Reasoning [16.01240703148773]
  We propose a new task called multihop question generation that asks complex and semantically relevant questions. To solve the problem, we propose multi-hop answer-focused reasoning on the grounded answer-centric entity graph.
  arXiv  Detail & Related papers  (2020-09-16T00:30:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.