CALRK-Bench: Evaluating Context-Aware Legal Reasoning in Korean Law

• URL: http://arxiv.org/abs/2603.26332v1
• Date: Fri, 27 Mar 2026 11:54:23 GMT
• Title: CALRK-Bench: Evaluating Context-Aware Legal Reasoning in Korean Law
• Abstract summary: We propose CALRK-Bench, a context-aware legal reasoning benchmark based on the legal system in Korean.<n>CALRK-Bench evaluates whether models can identify the temporal validity of legal norms, determine whether sufficient legal information is available for a given case, and understand the reasons behind shifts in legal judgments.
• Score: 6.44766672234951
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Legal reasoning requires not only the application of legal rules but also an understanding of the context in which those rules operate. However, existing legal benchmarks primarily evaluate rule application under the assumption of fixed norms, and thus fail to capture situations where legal judgments shift or where multiple norms interact. In this work, we propose CALRK-Bench, a context-aware legal reasoning benchmark based on the legal system in Korean. CALRK-Bench evaluates whether models can identify the temporal validity of legal norms, determine whether sufficient legal information is available for a given case, and understand the reasons behind shifts in legal judgments. The dataset is constructed from legal precedents and legal consultation records, and is validated by legal experts. Experimental results show that even recent large language models consistently exhibit low performance on these three tasks. CALRK-Bench provides a new stress test for evaluating context-aware legal reasoning rather than simple memorization of legal knowledge. Our code is available at https://github.com/jhCOR/CALRKBench.

Related papers

• The Measurement Gap in the Automation of EU Law: Benchmarking Doctrinal Legal Reasoning under the EU AI Act [0.08460698440162888]
  Large language models now produce legal text of at least median quality.<n>No existing benchmark can evaluate whether they perform doctrinal legal reasoning.<n>The EU AI Act makes "appropriate accuracy" a binding requirement for high-risk AI used in the judicial domain.
  arXiv  Detail & Related papers  (2026-06-16T16:57:12Z)
• Legal Reasoning Is Not Lawyering: Rethinking Legal Benchmarks for Pro Se Access to Justice [0.0]
  Legal AI benchmark research frequently invokes the assumption that large language models can improve access to justice.<n>We argue that current benchmarks are not equipped to support this assumption because they evaluate legal reasoning over inputs that have already been preprocessed by legal experts.<n>We conclude by calling for legal benchmarks that directly measure robustness under pro se-like inputs so that access-to-justice claims about legal AI can become empirically testable.
  arXiv  Detail & Related papers  (2026-06-16T14:19:04Z)
• WisdomInterrogatory (LuWen): An Open-Source Legal Large Language Model Technical Report [55.27414605169639]
  We present WisdomInterrogatory (LuWen), an open-source Chinese legal language model built upon the Baichuan foundation model.<n>We evaluate LuWen on five representative legal tasks spanning both prediction and generation settings.
  arXiv  Detail & Related papers  (2026-04-08T06:59:07Z)
• LegalOne: A Family of Foundation Models for Reliable Legal Reasoning [54.57434222018289]
  We present LegalOne, a family of foundational models specifically tailored for the Chinese legal domain.<n>LegalOne is developed through a comprehensive three-phase pipeline designed to master legal reasoning.<n>We publicly release the LegalOne weights and the LegalKit evaluation framework to advance the field of Legal AI.
  arXiv  Detail & Related papers  (2026-01-31T10:18:32Z)
• PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice [67.71760070255425]
  We introduce PLawBench, a practical benchmark for evaluating large language models (LLMs) in legal practice scenarios.<n>PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics.<n>Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2026-01-23T11:36:10Z)
• Capturing Legal Reasoning Paths from Facts to Law in Court Judgments using Knowledge Graphs [0.0]
  Court judgments reveal how legal rules have been interpreted and applied to facts.<n>Existing automated approaches for capturing legal reasoning do not accurately trace how facts relate to legal norms.<n>This paper builds a legal knowledge graph from 648 Japanese administrative court decisions.
  arXiv  Detail & Related papers  (2025-08-24T12:51:40Z)
• GLARE: Agentic Reasoning for Legal Judgment Prediction [60.13483016810707]
  Legal judgment prediction (LJP) has become increasingly important in the legal field.<n>Existing large language models (LLMs) have significant problems of insufficient reasoning due to a lack of legal knowledge.<n>We introduce GLARE, an agentic legal reasoning framework that dynamically acquires key legal knowledge by invoking different modules.
  arXiv  Detail & Related papers  (2025-08-22T13:38:12Z)
• LEXam: Benchmarking Legal Reasoning on 340 Law Exams [76.3521146499006]
  We introduce textscLEXam, a novel benchmark derived from 340 law exams spanning 116 law school courses across a range of subjects and degree levels.<n>The dataset comprises 4,886 law exam questions in English and German, including 2,841 long-form, open-ended questions and 2,045 multiple-choice questions.<n>Our results underscore the effectiveness of the dataset in differentiating between models with varying capabilities.
  arXiv  Detail & Related papers  (2025-05-19T08:48:12Z)
• LegalBench.PT: A Benchmark for Portuguese Law [17.554201334646056]
  We present LegalBench.PT, the first comprehensive legal benchmark covering key areas of Portuguese law.<n>We first collect long-form questions and answers from real law exams, and then use GPT-4o to convert them into multiple-choice, true/false, and matching formats.
  arXiv  Detail & Related papers  (2025-02-22T21:07:12Z)
• LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models [15.98468948605927]
  LegalBench is a benchmark consisting of 162 tasks covering six different types of legal reasoning. This paper describes LegalBench, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LegalBench enables.
  arXiv  Detail & Related papers  (2023-08-20T22:08:03Z)
• SAILER: Structure-aware Pre-trained Language Model for Legal Case Retrieval [75.05173891207214]
  Legal case retrieval plays a core role in the intelligent legal system. Most existing language models have difficulty understanding the long-distance dependencies between different structures. We propose a new Structure-Aware pre-traIned language model for LEgal case Retrieval.
  arXiv  Detail & Related papers  (2023-04-22T10:47:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.