LJ-Bench: Ontology-Based Benchmark for U.S. Crime

• URL: http://arxiv.org/abs/2603.20572v1
• Date: Sat, 21 Mar 2026 00:02:51 GMT
• Title: LJ-Bench: Ontology-Based Benchmark for U.S. Crime
• Abstract summary: Existing benchmarks only focus on a handful types of illegal activities, and are not grounded in legal works.<n>We introduce an ontology of crime-related concepts grounded in the legal frameworks of Model Panel Code.<n>This structured knowledge forms the foundation for LJ-Bench, the first comprehensive benchmark designed to evaluate LLM robustness against a wide range of illegal activities.
• Score: 13.90031857929218
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The potential of Large Language Models (LLMs) to provide harmful information remains a significant concern due to the vast breadth of illegal queries they may encounter. Unfortunately, existing benchmarks only focus on a handful types of illegal activities, and are not grounded in legal works. In this work, we introduce an ontology of crime-related concepts grounded in the legal frameworks of Model Panel Code, which serves as an influential reference for criminal law and has been adopted by many U.S. states, and instantiated using Californian Law. This structured knowledge forms the foundation for LJ-Bench, the first comprehensive benchmark designed to evaluate LLM robustness against a wide range of illegal activities. Spanning 76 distinct crime types organized taxonomically, LJ-Bench enables systematic assessment of diverse attacks, revealing valuable insights into LLM vulnerabilities across various crime categories: LLMs exhibit heightened susceptibility to attacks targeting societal harm rather than those directly impacting individuals. Our benchmark aims to facilitate the development of more robust and trustworthy LLMs. The LJ-Bench benchmark and LJ-Ontology, along with experiments implementation for reproducibility are publicly available at https://github.com/AndreaTseng/LJ-Bench.

Related papers

• Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts [0.25604314983611276]
  The Swiss Federal Supreme Court uses small on-premises models for tentative translations and short-passage summarization across the four official languages.<n>We introduce TF-RefusalBench, a benchmark for criminal-law translation and summarization derived from public Swiss Supreme Court rulings.<n>We then use TF-RefusalBench to show that over-alignment is a multifaceted phenomenon, influenced by the model and the prompt and text languages being processed.
  arXiv  Detail & Related papers  (2026-06-22T14:08:08Z)
• TriBench-Ko: Evaluating LLM Risks in Judicial Workflows [10.833772229369226]
  TriBench-Ko is a Korean benchmark designed to evaluate potential deployment risks of large language models.<n>It covers four core tasks: summarization, precedent retrieval, legal issue extraction, and evidence analysis.<n>It jointly assesses model behavior across multiple deployment risk categories.
  arXiv  Detail & Related papers  (2026-05-05T14:20:28Z)
• PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice [67.71760070255425]
  We introduce PLawBench, a practical benchmark for evaluating large language models (LLMs) in legal practice scenarios.<n>PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics.<n>Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2026-01-23T11:36:10Z)
• LexGenius: An Expert-Level Benchmark for Large Language Models in Legal General Intelligence [74.05988707492058]
  Legal general intelligence (GI) refers to artificial intelligence (AI) that encompasses legal understanding, reasoning, and decision-making.<n>Existing benchmarks are result-oriented and fail to systematically evaluate the legal intelligence of large language models (LLMs)<n>We propose LexGenius, an expert-level Chinese legal benchmark for evaluating legal GI in LLMs.
  arXiv  Detail & Related papers  (2025-12-04T08:48:02Z)
• Large Language Models' Complicit Responses to Illicit Instructions across Socio-Legal Contexts [54.15982476754607]
  Large language models (LLMs) are now deployed at unprecedented scale, assisting millions of users in daily tasks.<n>This study defines complicit facilitation as the provision of guidance or support that enables illicit user instructions.<n>Using real-world legal cases and established legal frameworks, we construct an evaluation benchmark spanning 269 illicit scenarios and 50 illicit intents.
  arXiv  Detail & Related papers  (2025-11-25T16:01:31Z)
• Better Call CLAUSE: A Discrepancy Benchmark for Auditing LLMs Legal Reasoning Capabilities [15.35489310097019]
  CLAUSE is a first-of-its-kind benchmark designed to evaluate the fragility of an LLM's legal reasoning.<n>Our work outlines a path to identify and correct such reasoning failures in legal AI.
  arXiv  Detail & Related papers  (2025-11-01T00:51:21Z)
• PRISON: Unmasking the Criminal Potential of Large Language Models [25.210177069866656]
  We quantify large language models' criminal potential across five traits: False Statements, Frame-Up, Psychological Manipulation, Emotional Disguise, and Moral Disengagement.<n>Results show that state-of-the-art LLMs frequently exhibit emergent criminal tendencies, such as proposing misleading statements or evasion tactics.<n>When placed in a detective role, models recognize deceptive behavior with only 44% accuracy on average, revealing a striking mismatch between conducting and detecting criminal behavior.
  arXiv  Detail & Related papers  (2025-06-19T09:06:27Z)
• J&H: Evaluating the Robustness of Large Language Models Under Knowledge-Injection Attacks in Legal Domain [12.550611136062722]
  We propose a method of legal knowledge injection attacks for robustness testing.<n>The aim of the framework is to explore whether LLMs perform deductive reasoning when accomplishing legal tasks.<n>We have collected mistakes that legal experts might make in judicial decisions in the real world.
  arXiv  Detail & Related papers  (2025-03-24T05:42:05Z)
• LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models [17.90483181611453]
  Large language models (LLMs) have made significant progress in natural language processing tasks and demonstrate considerable potential in the legal domain. Applying existing LLMs to legal systems without careful evaluation of their potential and limitations could pose significant risks in legal practice. We introduce a standardized comprehensive Chinese legal benchmark LexEval.
  arXiv  Detail & Related papers  (2024-09-30T13:44:00Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics.<n>Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching.<n>Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming [64.86326523181553]
  ALERT is a large-scale benchmark to assess safety based on a novel fine-grained risk taxonomy. It aims to identify vulnerabilities, inform improvements, and enhance the overall safety of the language models.
  arXiv  Detail & Related papers  (2024-04-06T15:01:47Z)
• A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction [60.70089334782383]
  Large language models (LLMs) have demonstrated great potential for domain-specific applications. Recent disputes over GPT-4's law evaluation raise questions concerning their performance in real-world legal tasks. We design practical baseline solutions based on LLMs and test on the task of legal judgment prediction.
  arXiv  Detail & Related papers  (2023-10-18T07:38:04Z)
• LawBench: Benchmarking Legal Knowledge of Large Language Models [35.2812008533622]
  Large language models (LLMs) have demonstrated strong capabilities in various aspects. It is unclear how much legal knowledge they possess and whether they can reliably perform legal-related tasks. LawBench has been meticulously crafted to have precise assessment of the LLMs' legal capabilities from three cognitive levels.
  arXiv  Detail & Related papers  (2023-09-28T09:35:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.