Towards Unsupervised Question Answering System with Multi-level Summarization for Legal Text

• URL: http://arxiv.org/abs/2403.13107v2
• Date: Tue, 2 Jul 2024 03:35:53 GMT
• Title: Towards Unsupervised Question Answering System with Multi-level Summarization for Legal Text
• Authors: M Manvith Prabhu, Haricharana Srinivasa, Anand Kumar M,
• Abstract summary: This paper summarizes Team SCaLAR's work on SemEval-2024 Task 5: Legal Argument Reasoning in Civil Procedure. We propose a simple yet novel similarity and distance-based unsupervised approach to generate labels. Our unsupervised system witnessed a 20-point increase in macro F1-score on the development set and a 10-point increase on the test set.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper summarizes Team SCaLAR's work on SemEval-2024 Task 5: Legal Argument Reasoning in Civil Procedure. To address this Binary Classification task, which was daunting due to the complexity of the Legal Texts involved, we propose a simple yet novel similarity and distance-based unsupervised approach to generate labels. Further, we explore the Multi-level fusion of Legal-Bert embeddings using ensemble features, including CNN, GRU, and LSTM. To address the lengthy nature of Legal explanation in the dataset, we introduce T5-based segment-wise summarization, which successfully retained crucial information, enhancing the model's performance. Our unsupervised system witnessed a 20-point increase in macro F1-score on the development set and a 10-point increase on the test set, which is promising given its uncomplicated architecture.

Related papers

• Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning [19.477062052536887]
  We propose the Logical-Semantic Integration Model (LSIM), a supervised framework that bridges semantic and logical coherence. LSIM comprises three components: reinforcement learning predicts a structured fact-rule chain for each question, a trainable Deep Structured Semantic Model (DSSM) retrieves the most relevant candidate questions and in-answer learning generates the final answer. Our experiments on a real-world legal dataset QA-validated through both automated metrics and human evaluation-demonstrate that LSIM significantly enhances accuracy and reliability compared to existing methods.
  arXiv  Detail & Related papers  (2025-02-11T19:33:07Z)
• LegalSeg: Unlocking the Structure of Indian Legal Judgments Through Rhetorical Role Classification [6.549338652948716]
  We introduce LegalSeg, the largest annotated dataset for this task, comprising over 7,000 documents and 1.4 million sentences, labeled with 7 rhetorical roles. Our results demonstrate that models incorporating broader context, structural relationships, and sequential sentence information outperform those relying solely on sentence-level features.
  arXiv  Detail & Related papers  (2025-02-09T10:07:05Z)
• Multi-Agent Simulator Drives Language Models for Legal Intensive Interaction [37.856194200684364]
  This paper introduces a Multi-agent Legal Simulation Driver (MASER) to scalably generate synthetic data by simulating interactive legal scenarios. MASER ensures the consistency of legal attributes between participants and introduces a supervisory mechanism to align participants' characters and behaviors.
  arXiv  Detail & Related papers  (2025-02-08T15:05:24Z)
• Textualized Agent-Style Reasoning for Complex Tasks by Multiple Round LLM Generation [49.27250832754313]
  We present AgentCOT, a llm-based autonomous agent framework. At each step, AgentCOT selects an action and executes it to yield an intermediate result with supporting evidence. We introduce two new strategies to enhance the performance of AgentCOT.
  arXiv  Detail & Related papers  (2024-09-19T02:20:06Z)
• InternLM-Law: An Open Source Chinese Legal Large Language Model [72.2589401309848]
  InternLM-Law is a specialized LLM tailored for addressing diverse legal queries related to Chinese laws. We meticulously construct a dataset in the Chinese legal domain, encompassing over 1 million queries. InternLM-Law achieves the highest average performance on LawBench, outperforming state-of-the-art models, including GPT-4, on 13 out of 20 subtasks.
  arXiv  Detail & Related papers  (2024-06-21T06:19:03Z)
• Exploring Large Language Models and Hierarchical Frameworks for Classification of Large Unstructured Legal Documents [0.6349503549199403]
  We explore the classification of large legal documents and their lack of structural information with a deep-learning-based hierarchical framework. Specifically, we divide a document into parts to extract their embeddings from the last four layers of a custom fine-tuned Large Language Model. Our approach achieves a minimum total performance gain of approximately 2 points over previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-03-11T16:24:08Z)
• Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy [48.29181662640212]
  Factual inconsistencies pose a significant hurdle for the faithful summarization by generative models. We consolidate key error types of inconsistent facts in summaries, and incorporate them to facilitate both the zero-shot and supervised paradigms of LLMs.
  arXiv  Detail & Related papers  (2024-02-20T08:41:23Z)
• A Hierarchical Neural Framework for Classification and its Explanation in Large Unstructured Legal Documents [0.5812284760539713]
  We define this problem as "scarce annotated legal documents" We propose a deep-learning-based classification framework which we call MESc. We also propose an explanation extraction algorithm named ORSE.
  arXiv  Detail & Related papers  (2023-09-19T12:18:28Z)
• Pushing the Limits of ChatGPT on NLP Tasks [79.17291002710517]
  Despite the success of ChatGPT, its performances on most NLP tasks are still well below the supervised baselines. In this work, we looked into the causes, and discovered that its subpar performance was caused by the following factors. We propose a collection of general modules to address these issues, in an attempt to push the limits of ChatGPT on NLP tasks.
  arXiv  Detail & Related papers  (2023-06-16T09:40:05Z)
• Conversational Semantic Parsing [50.954321571100294]
  Session-based properties such as co-reference resolution and context carryover are processed downstream in a pipelined system. We release a new session-based, compositional task-oriented parsing dataset of 20k sessions consisting of 60k utterances. We propose a new family of Seq2Seq models for the session-based parsing above, which achieve better or comparable performance to the current state-of-the-art on ATIS, SNIPS, TOP and DSTC2.
  arXiv  Detail & Related papers  (2020-09-28T22:08:00Z)
• Extractive Summarization as Text Matching [123.09816729675838]
  This paper creates a paradigm shift with regard to the way we build neural extractive summarization systems. We formulate the extractive summarization task as a semantic text matching problem. We have driven the state-of-the-art extractive result on CNN/DailyMail to a new level (44.41 in ROUGE-1)
  arXiv  Detail & Related papers  (2020-04-19T08:27:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.