Comparison of Unsupervised Metrics for Evaluating Judicial Decision Extraction

• URL: http://arxiv.org/abs/2510.01792v1
• Date: Thu, 02 Oct 2025 08:32:16 GMT
• Title: Comparison of Unsupervised Metrics for Evaluating Judicial Decision Extraction
• Abstract summary: This study evaluates 16 unsupervised metrics, including novel formulations, to assess the quality of extracting seven semantic blocks from 1,000 Russian judicial decisions.<n>These metrics, spanning document-based, semantic, structural, pseudo-ground truth, and legal-specific categories, operate without pre-annotated ground truth.<n>This work advances legal NLP by providing annotation-free evaluation tools, with implications for judicial analytics and ethical AI deployment.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid advancement of artificial intelligence in legal natural language processing demands scalable methods for evaluating text extraction from judicial decisions. This study evaluates 16 unsupervised metrics, including novel formulations, to assess the quality of extracting seven semantic blocks from 1,000 anonymized Russian judicial decisions, validated against 7,168 expert reviews on a 1--5 Likert scale. These metrics, spanning document-based, semantic, structural, pseudo-ground truth, and legal-specific categories, operate without pre-annotated ground truth. Bootstrapped correlations, Lin's concordance correlation coefficient (CCC), and mean absolute error (MAE) reveal that Term Frequency Coherence (Pearson $r = 0.540$, Lin CCC = 0.512, MAE = 0.127) and Coverage Ratio/Block Completeness (Pearson $r = 0.513$, Lin CCC = 0.443, MAE = 0.139) best align with expert ratings, while Legal Term Density (Pearson $r = -0.479$, Lin CCC = -0.079, MAE = 0.394) show strong negative correlations. The LLM Evaluation Score (mean = 0.849, Pearson $r = 0.382$, Lin CCC = 0.325, MAE = 0.197) showed moderate alignment, but its performance, using gpt-4.1-mini via g4f, suggests limited specialization for legal textse. These findings highlight that unsupervised metrics, including LLM-based approaches, enable scalable screening but, with moderate correlations and low CCC values, cannot fully replace human judgment in high-stakes legal contexts. This work advances legal NLP by providing annotation-free evaluation tools, with implications for judicial analytics and ethical AI deployment.

Related papers

• SAGE: State-Grounded, Abstention-Aware Evaluation of Task-Oriented Dialogue Agents [2.853407724009394]
  SAGE compiles a workflow specification and per-turn state diff into atomic, schema-grounded criteria.<n>SAGE-Core decides 81--91% of criteria with only the compiler, symbolic rules, and on-device encoders.<n>SAGE-LLM adds an optional focused-LLM fallback for open-class criteria.
  arXiv  Detail & Related papers  (2026-08-31T22:14:52Z)
• Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability [28.15461581779256]
  We develop a method for estimating correlation-based reliability metrics of LLM judges from limited annotations.<n>Metric Match selects a subset of samples for human annotation such that the subset matches the population reliability metric.<n>We empirically show that Metric Match achieves a win-rate of 0.838 against random subset selection.
  arXiv  Detail & Related papers  (2026-06-12T23:54:16Z)
• Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization [0.0]
  We show that grounding evaluation in authorship verification theory transforms what benchmarks can measure.<n>We evaluate four inference-time personalization methods across 50 authors and 1,000 generations.
  arXiv  Detail & Related papers  (2026-04-29T09:17:01Z)
• Understanding LLM Evaluator Behavior: A Structured Multi-Evaluator Framework for Merchant Risk Assessment [26.786161923794115]
  Large Language Models (LLMs) are increasingly used as evaluators of reasoning quality, yet their reliability and bias in payments-risk settings remain poorly understood.<n>We introduce a structured multi-evaluator framework for assessing LLM reasoning in Merchant Category Code (MCC)-based merchant risk assessment.
  arXiv  Detail & Related papers  (2026-02-04T22:55:16Z)
• Legal experts disagree with rationale extraction techniques for explaining ECtHR case outcome classification [9.334783986218232]
  Interpretability is critical for applications of large language models in the legal domain.<n>We propose a comparative analysis framework for model-agnostic interpretability techniques.<n>We show that the model's "reasons" for predicting a violation differ substantially from those of legal experts.
  arXiv  Detail & Related papers  (2026-01-18T14:03:17Z)
• Knowing But Not Doing: Convergent Morality and Divergent Action in LLMs [18.492825007258656]
  We present ValAct-15k, a dataset of 3,000 advice-seeking scenarios derived from Reddit.<n>We find near-perfect cross-model consistency in scenario-based decisions.<n>Humans and Large Language Models show weak correspondence between self-reported and enacted values.
  arXiv  Detail & Related papers  (2026-01-12T20:07:30Z)
• Relative Scaling Laws for LLMs [91.73497548097775]
  Scaling laws describe how language models improve with additional data, parameters, and compute.<n>We introduce relative scaling laws, which track how performance gaps between test distributions evolve with scale.<n>These results show that although scaling improves overall performance, it is not a universal equalizer.
  arXiv  Detail & Related papers  (2025-10-28T16:55:22Z)
• ObjexMT: Objective Extraction and Metacognitive Calibration for LLM-as-a-Judge under Multi-Turn Jailbreaks [12.396822247035578]
  claude-sonnet-4 attains the best objective-extraction accuracy (0.515) and calibration (ECE 0.296; Brier 0.324)<n>We recommend exposing objectives when feasible and gating decisions by confidence otherwise.
  arXiv  Detail & Related papers  (2025-08-23T03:32:04Z)
• Let's Measure Information Step-by-Step: LLM-Based Evaluation Beyond Vibes [14.371259136517802]
  We study robustness of AI systems without ground truth by exploiting a link between strategic gaming and information loss.<n>We analyze which information-theoretic mechanisms resist adversarial bounds, extending finite-sample manipulation to show that bounded f-divergences maintain under attacks.
  arXiv  Detail & Related papers  (2025-08-07T15:11:43Z)
• Claim Extraction for Fact-Checking: Data, Models, and Automated Metrics [0.0]
  We release the FEVERFact dataset, with 17K atomic factual claims extracted from 4K contextualised Wikipedia sentences.<n>For each metric, we implement a scale using a reduction to an already-explored NLP task.<n>We validate our metrics against human grading of generic claims, to see that the model ranking on $F_fact$, our hardest metric, did not change.
  arXiv  Detail & Related papers  (2025-02-07T14:20:45Z)
• Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs) [0.5434005537854512]
  This study explored the potential of large language models (LLMs) to automate OSCE evaluations using the Master Interview Rating Scale (MIRS)<n>We compared the performance of four state-of-the-art LLMs in evaluating OSCE transcripts across all 28 items of the MIRS under the conditions of zero-shot, chain-of-thought (CoT), few-shot, and multi-step prompting.
  arXiv  Detail & Related papers  (2025-01-21T04:05:45Z)
• Assessing the Sensitivity and Alignment of FOL Closeness Metrics [10.795521518273214]
  We study the sensitivity of existing NL-, FOL-, and graph-based metrics to capture differences between a sampled FOL and its corresponding ground-truth.<n>We show that combining metrics enhances both robustness and sensitivity compared to using individual metrics.
  arXiv  Detail & Related papers  (2025-01-15T06:22:35Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• LLMs Can Patch Up Missing Relevance Judgments in Evaluation [56.51461892988846]
  We use large language models (LLMs) to automatically label unjudged documents. We simulate scenarios with varying degrees of holes by randomly dropping relevant documents from the relevance judgment in TREC DL tracks. Our method achieves a Kendall tau correlation of 0.87 and 0.92 on an average for Vicuna-7B and GPT-3.5 Turbo respectively.
  arXiv  Detail & Related papers  (2024-05-08T00:32:19Z)
• Uncertainty in Language Models: Assessment through Rank-Calibration [65.10149293133846]
  Language Models (LMs) have shown promising performance in natural language generation. It is crucial to correctly quantify their uncertainty in responding to given inputs. We develop a novel and practical framework, termed $Rank$-$Calibration$, to assess uncertainty and confidence measures for LMs.
  arXiv  Detail & Related papers  (2024-04-04T02:31:05Z)
• Cobra Effect in Reference-Free Image Captioning Metrics [58.438648377314436]
  A proliferation of reference-free methods, leveraging visual-language pre-trained models (VLMs), has emerged. In this paper, we study if there are any deficiencies in reference-free metrics. We employ GPT-4V as an evaluative tool to assess generated sentences and the result reveals that our approach achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2024-02-18T12:36:23Z)
• Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References [123.39034752499076]
  Div-Ref is a method to enhance evaluation benchmarks by enriching the number of references. We conduct experiments to empirically demonstrate that diversifying the expression of reference can significantly enhance the correlation between automatic evaluation and human evaluation.
  arXiv  Detail & Related papers  (2023-05-24T11:53:29Z)
• Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis [79.18261352971284]
  We introduce SESCORE, a model-based metric that is highly correlated with human judgements without requiring human annotation. We evaluate SESCORE against existing metrics by comparing how their scores correlate with human ratings. SESCORE even achieves comparable performance to the best supervised metric COMET, despite receiving no human-annotated training data.
  arXiv  Detail & Related papers  (2022-10-10T22:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.