STARD: A Chinese Statute Retrieval Dataset with Real Queries Issued by Non-professionals

• URL: http://arxiv.org/abs/2406.15313v1
• Date: Fri, 21 Jun 2024 17:10:09 GMT
• Title: STARD: A Chinese Statute Retrieval Dataset with Real Queries Issued by Non-professionals
• Authors: Weihang Su, Yiran Hu, Anzhe Xie, Qingyao Ai, Zibing Que, Ning Zheng, Yun Liu, Weixing Shen, Yiqun Liu,
• Abstract summary: Statute retrieval aims to find relevant statutory articles for specific queries. Existing statute retrieval benchmarks focus on formal and professional queries from sources like bar exams and legal case documents. To address this gap, we introduce the STAtute Retrieval dataset (STARD) Unlike existing statute retrieval datasets, STARD captures the complexity and diversity of real queries from the general public.
• Score: 14.002280587675175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Statute retrieval aims to find relevant statutory articles for specific queries. This process is the basis of a wide range of legal applications such as legal advice, automated judicial decisions, legal document drafting, etc. Existing statute retrieval benchmarks focus on formal and professional queries from sources like bar exams and legal case documents, thereby neglecting non-professional queries from the general public, which often lack precise legal terminology and references. To address this gap, we introduce the STAtute Retrieval Dataset (STARD), a Chinese dataset comprising 1,543 query cases collected from real-world legal consultations and 55,348 candidate statutory articles. Unlike existing statute retrieval datasets, which primarily focus on professional legal queries, STARD captures the complexity and diversity of real queries from the general public. Through a comprehensive evaluation of various retrieval baselines, we reveal that existing retrieval approaches all fall short of these real queries issued by non-professional users. The best method only achieves a Recall@100 of 0.907, suggesting the necessity for further exploration and additional research in this area. All the codes and datasets are available at: https://github.com/oneal2000/STARD/tree/main

Related papers

• JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking [81.88787401178378]
  We introduce JudgeRank, a novel agentic reranker that emulates human cognitive processes when assessing document relevance. We evaluate JudgeRank on the reasoning-intensive BRIGHT benchmark, demonstrating substantial performance improvements over first-stage retrieval methods. In addition, JudgeRank performs on par with fine-tuned state-of-the-art rerankers on the popular BEIR benchmark, validating its zero-shot generalization capability.
  arXiv  Detail & Related papers  (2024-10-31T18:43:12Z)
• Exploiting LLMs' Reasoning Capability to Infer Implicit Concepts in Legal Information Retrieval [6.952344923975001]
  This work focuses on utilizing the logical reasoning capabilities of large language models (LLMs) to identify relevant legal terms. The proposed retrieval system integrates additional information from the term--based expansion and query reformulation to improve the retrieval accuracy. Experiments on COLIEE 2022 and COLIEE 2023 datasets show that extra knowledge from LLMs helps to improve the retrieval result of both lexical and semantic ranking models.
  arXiv  Detail & Related papers  (2024-10-16T01:34:14Z)
• Enhancing Legal Case Retrieval via Scaling High-quality Synthetic Query-Candidate Pairs [67.54302101989542]
  Legal case retrieval aims to provide similar cases as references for a given fact description. Existing works mainly focus on case-to-case retrieval using lengthy queries. Data scale is insufficient to satisfy the training requirements of existing data-hungry neural models.
  arXiv  Detail & Related papers  (2024-10-09T06:26:39Z)
• Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice [32.550204238857724]
  We propose a human-centric legal NLP pipeline, covering data sourcing, inference, and evaluation. We release a dataset, LegalQA, with real and specific legal questions spanning from employment law to criminal law. We show that retrieval-augmented generation from only 850 citations in the train set can match or outperform internet-wide retrieval.
  arXiv  Detail & Related papers  (2024-09-12T02:40:28Z)
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval [54.54576644403115]
  Many complex real-world queries require in-depth reasoning to identify relevant documents. We introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding.
  arXiv  Detail & Related papers  (2024-07-16T17:58:27Z)
• InternLM-Law: An Open Source Chinese Legal Large Language Model [72.2589401309848]
  InternLM-Law is a specialized LLM tailored for addressing diverse legal queries related to Chinese laws. We meticulously construct a dataset in the Chinese legal domain, encompassing over 1 million queries. InternLM-Law achieves the highest average performance on LawBench, outperforming state-of-the-art models, including GPT-4, on 13 out of 20 subtasks.
  arXiv  Detail & Related papers  (2024-06-21T06:19:03Z)
• MUSER: A Multi-View Similar Case Retrieval Dataset [65.36779942237357]
  Similar case retrieval (SCR) is a representative legal AI application that plays a pivotal role in promoting judicial fairness. Existing SCR datasets only focus on the fact description section when judging the similarity between cases. We present M, a similar case retrieval dataset based on multi-view similarity measurement and comprehensive legal element with sentence-level legal element annotations.
  arXiv  Detail & Related papers  (2023-10-24T08:17:11Z)
• An Intent Taxonomy of Legal Case Retrieval [43.22489520922202]
  Legal case retrieval is a special Information Retrieval(IR) task focusing on legal case documents. We present a novel hierarchical intent taxonomy of legal case retrieval. We reveal significant differences in user behavior and satisfaction under different search intents in legal case retrieval.
  arXiv  Detail & Related papers  (2023-07-25T07:27:32Z)
• SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval [75.05173891207214]
  Legal case retrieval plays a core role in the intelligent legal system. Most existing language models have difficulty understanding the long-distance dependencies between different structures. We propose a new Structure-Aware pre-traIned language model for LEgal case Retrieval.
  arXiv  Detail & Related papers  (2023-04-22T10:47:01Z)
• Case law retrieval: problems, methods, challenges and evaluations in the last 20 years [23.13408774493739]
  We survey methods for case law retrieval from the past 20 years. We outline the problems and challenges facing evaluation of case law retrieval systems going forward.
  arXiv  Detail & Related papers  (2022-02-15T06:01:36Z)
• A Statutory Article Retrieval Dataset in French [4.082216579462797]
  We introduce the Belgian Statutory Article Retrieval dataset (BSARD) BSARD consists of 1,100+ French native legal questions labeled by experienced jurists with relevant articles from a corpus of 22,600+ Belgian law articles. We benchmark several unsupervised information retrieval methods based on term weighting and pooled embeddings. Our best performing baseline achieves 50.8% R@100, which is promising for the feasibility of the task and indicates that there is still substantial room for improvement.
  arXiv  Detail & Related papers  (2021-08-26T13:50:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.