Toward Human Rights Benchmarking for LLMs: A Pilot Methodology

• URL: http://arxiv.org/abs/2608.10268v1
• Date: Mon, 10 Aug 2026 22:09:05 GMT
• Title: Toward Human Rights Benchmarking for LLMs: A Pilot Methodology
• Abstract summary: HumRightsBench is the first expert-validated, scenario-based benchmark for evaluating reasoning grounded in the obligation structure of human rights law.<n>We find that model accuracy scores range considerably across legal reasoning tasks.
• Score: 0.002301825353472044
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language models (LLMs) increasingly mediate legal determinations over what human rights are realized, and how. Yet, no evaluation benchmark exists to assess whether they can reason correctly about human rights law. To this end, we report our efforts to develop a robust and scalable methodology for creating HumRightsBench: the first expert-validated, scenario-based benchmark for evaluating reasoning grounded in the obligation structure of international human rights law. We adapt the IRAC framework for legal reasoning to better suit the unique reasoning patterns of human rights work (substituting P, "proposing remedies," for C, "legal conclusion," yielding IRAP) to structure our evaluation heuristics. We also produce a pilot series of authentic scenarios designed to implicate the many dimensions of real-world human rights issues and annotated by human rights lawyers and professionals across the world. Ultimately, we find that model accuracy scores range considerably across legal reasoning tasks (overall model performance ranges from 0.339 to 0.577, task min-max ranges from 0.025 to 0.774), which strongly implies that HumRightsBench is a capable instrument for advancing this emerging subfield of AI evaluations science at a critical moment in its evolution.

Related papers

• Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases [17.869207928898042]
  We investigate how LLMs reason in the context of legal case forecasting.<n>We use legal cases from the European Court of Human Rights as a testbed.<n>We find that the examined model scores far from ideal in legal reasoning.
  arXiv  Detail & Related papers  (2026-08-17T22:11:22Z)
• When Do Language Models Endorse Limitations on Human Rights Principles? [82.84306700922664]
  We evaluate how Large Language Models (LLMs) navigate trade-offs involving the Universal Declaration of Human Rights (UDHR)<n>Our analysis of eleven major LLMs reveals systematic biases where models accept limiting Economic, Social, and Cultural rights more often than Political and Civil rights.
  arXiv  Detail & Related papers  (2026-03-04T16:01:53Z)
• LegalOne: A Family of Foundation Models for Reliable Legal Reasoning [54.57434222018289]
  We present LegalOne, a family of foundational models specifically tailored for the Chinese legal domain.<n>LegalOne is developed through a comprehensive three-phase pipeline designed to master legal reasoning.<n>We publicly release the LegalOne weights and the LegalKit evaluation framework to advance the field of Legal AI.
  arXiv  Detail & Related papers  (2026-01-31T10:18:32Z)
• PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice [67.71760070255425]
  We introduce PLawBench, a practical benchmark for evaluating large language models (LLMs) in legal practice scenarios.<n>PLawBench comprises 850 questions across 13 practical legal scenarios, with each question accompanied by expert-designed evaluation rubrics.<n>Using an LLM-based evaluator aligned with human expert judgments, we evaluate 10 state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2026-01-23T11:36:10Z)
• ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India [10.522785783474857]
  We introduce Reinforcement Learning-based Legal Reasoning (ReGal), a framework that integrates Multi-Task Instruction Tuning with Reinforcement Learning from AI Feedback.<n>Our approach is evaluated across two critical legal tasks: (i) Court Judgment Prediction and Explanation (CJPE), and (ii) Legal Document Summarization.
  arXiv  Detail & Related papers  (2025-12-19T19:13:41Z)
• Assessing Human Rights Risks in AI: A Framework for Model Evaluation [0.10195618602298682]
  We contribute to the field of algorithmic auditing by presenting a framework to computationally assess human rights risk.<n>We develop an approach to evaluating a model to make grounded claims about the level of risk a model poses to particular human rights.<n>Because a human rights approach centers on real-world harms, it requires evaluating AI systems in the specific contexts in which they are deployed.
  arXiv  Detail & Related papers  (2025-10-07T02:12:56Z)
• ClaimGen-CN: A Large-scale Chinese Dataset for Legal Claim Generation [56.79698529022327]
  Legal claims refer to the plaintiff's demands in a case and are essential to guiding judicial reasoning and case resolution.<n>This paper explores the problem of legal claim generation based on the given case's facts.<n>We construct ClaimGen-CN, the first dataset for Chinese legal claim generation task.
  arXiv  Detail & Related papers  (2025-08-24T07:19:25Z)
• An evidence-based methodology for human rights impact assessment (HRIA) in the development of AI data-intensive systems [49.1574468325115]
  We show that human rights already underpin the decisions in the field of data use. This work presents a methodology and a model for a Human Rights Impact Assessment (HRIA) The proposed methodology is tested in concrete case-studies to prove its feasibility and effectiveness.
  arXiv  Detail & Related papers  (2024-07-30T16:27:52Z)
• CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility [62.74405775089802]
  We present CValues, the first Chinese human values evaluation benchmark to measure the alignment ability of LLMs. As a result, we have manually collected adversarial safety prompts across 10 scenarios and induced responsibility prompts from 8 domains. Our findings suggest that while most Chinese LLMs perform well in terms of safety, there is considerable room for improvement in terms of responsibility.
  arXiv  Detail & Related papers  (2023-07-19T01:22:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.