From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text

• URL: http://arxiv.org/abs/2604.16270v1
• Date: Fri, 17 Apr 2026 17:28:23 GMT
• Title: From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text
• Abstract summary: This paper introduces a comprehensive dual-aspect evaluation framework to address this need.<n>We establish a performance benchmark for four state-of-the-art large language models across three key dimensions: Accuracy, Readability, and Consistency.<n>Our results reveal a crucial trade-off: models like Grok-1 excel in Readability and Consistency but compromise on fine-grained legal Accuracy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The complexity of Vietnam's legal texts presents a significant barrier to public access to justice. While Large Language Models offer a promising solution for legal text simplification, evaluating their true capabilities requires a multifaceted approach that goes beyond surface-level metrics. This paper introduces a comprehensive dual-aspect evaluation framework to address this need. First, we establish a performance benchmark for four state-of-the-art large language models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1) across three key dimensions: Accuracy, Readability, and Consistency. Second, to understand the "why" behind these performance scores, we conduct a large-scale error analysis on a curated dataset of 60 complex Vietnamese legal articles, using a novel, expert-validated error typology. Our results reveal a crucial trade-off: models like Grok-1 excel in Readability and Consistency but compromise on fine-grained legal Accuracy, while models like Claude 3 Opus achieve high Accuracy scores that mask a significant number of subtle but critical reasoning errors. The error analysis pinpoints \textit{Incorrect Example} and \textit{Misinterpretation} as the most prevalent failures, confirming that the primary challenge for current LLMs is not summarization but controlled, accurate legal reasoning. By integrating a quantitative benchmark with a qualitative deep dive, our work provides a holistic and actionable assessment of LLMs for legal applications.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• Achieving Text-based Person Retrieval with Any Granularity [80.69427598925718]
  This paper introduces a new paradigm, Text-based Person Retrieval with Any Granularity.<n>We formalize a five-level granularity spectrum and construct UFine6926-MG, a high-quality multi-grained annotated dataset.<n>We then propose the Cross-modal Multi-grained Aligning and Matching framework.
  arXiv  Detail & Related papers  (2026-07-23T08:42:25Z)
• ViLegalNLI: Natural Language Inference for Vietnamese Legal Texts [5.44919589318124]
  We introduce ViLegalNLI, the first large-scale Vietnamese Natural Language Inference dataset specifically constructed for the legal domain.<n>The dataset consists of 42,012 premise-hypothesis pairs derived from official statutory documents and annotated with binary inference labels.<n>We conduct extensive experiments on the ViLegalNLI using multilingual models, Vietnamese-specific pretrained language models, and instruction-tuned large language models.
  arXiv  Detail & Related papers  (2026-04-30T18:16:14Z)
• SoK: Large Language Model Copyright Auditing via Fingerprinting [69.14570598973195]
  We introduce a unified framework and formal taxonomy that categorizes existing methods into white-box and black-box approaches.<n>We propose LeaFBench, the first systematic benchmark for evaluating LLM fingerprinting under realistic deployment scenarios.
  arXiv  Detail & Related papers  (2025-08-27T12:56:57Z)
• LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text [0.0]
  We develop a regression model to evaluate the quality of legal texts based on clarity, coherence, and terminology.<n>Our analysis identifies three key findings: First, model quality levels off at 14 billion parameters, with only a marginal improvement of $2.7%$ noted at 72 billion parameters.
  arXiv  Detail & Related papers  (2025-05-30T17:30:18Z)
• Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models [11.379764847748378]
  Large language models (LLMs) often uncritically accept flawed or contradictory premises, leading to inefficient reasoning and unreliable outputs.<n>This emphasizes the significance of possessing the textbfPremise Critique Ability for LLMs, defined as the capacity to proactively identify and articulate errors in input premises.<n>We introduce the textbfPremise Critique Bench (PCBench), designed by incorporating four error types across three difficulty levels, paired with multi-faceted evaluation metrics.
  arXiv  Detail & Related papers  (2025-05-29T17:49:44Z)
• Refining Financial Consumer Complaints through Multi-Scale Model Interaction [8.504311452987036]
  This paper explores the task of legal text refinement that transforms informal, conversational inputs into persuasive legal arguments.<n>We introduce FinDR, a Chinese dataset of financial dispute records, annotated with official judgments on claim reasonableness.<n> Experimental results demonstrate that Multi-Scale Model Interaction (MSMI) significantly outperforms single-pass prompting strategies.
  arXiv  Detail & Related papers  (2025-04-14T05:51:31Z)
• Improving the Accuracy and Efficiency of Legal Document Tagging with Large Language Models and Instruction Prompts [0.6554326244334866]
  Legal-LLM is a novel approach that leverages the instruction-following capabilities of Large Language Models (LLMs) through fine-tuning.<n>We evaluate our method on two benchmark datasets, POSTURE50K and EURLEX57K, using micro-F1 and macro-F1 scores.
  arXiv  Detail & Related papers  (2025-04-12T18:57:04Z)
• Aplicação de Large Language Models na Análise e Síntese de Documentos Jurídicos: Uma Revisão de Literatura [0.0]
  Large Language Models (LLMs) have been increasingly used to optimize the analysis and synthesis of legal documents.<n>This study aims to conduct a systematic literature review to identify the state of the art in prompt engineering applied to LLMs in the legal context.
  arXiv  Detail & Related papers  (2025-04-01T12:34:00Z)
• Towards General Visual-Linguistic Face Forgery Detection(V2) [90.6600794602029]
  Face manipulation techniques have achieved significant advances, presenting serious challenges to security and social trust.<n>Recent works demonstrate that leveraging multimodal models can enhance the generalization and interpretability of face forgery detection.<n>We propose Face Forgery Text Generator (FFTG), a novel annotation pipeline that generates accurate text descriptions by leveraging forgery masks for initial region and type identification.
  arXiv  Detail & Related papers  (2025-02-28T04:15:36Z)
• JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking [81.88787401178378]
  We introduce JudgeRank, a novel agentic reranker that emulates human cognitive processes when assessing document relevance. We evaluate JudgeRank on the reasoning-intensive BRIGHT benchmark, demonstrating substantial performance improvements over first-stage retrieval methods. In addition, JudgeRank performs on par with fine-tuned state-of-the-art rerankers on the popular BEIR benchmark, validating its zero-shot generalization capability.
  arXiv  Detail & Related papers  (2024-10-31T18:43:12Z)
• Localizing Factual Inconsistencies in Attributable Text Generation [74.11403803488643]
  We introduce QASemConsistency, a new formalism for localizing factual inconsistencies in attributable text generation.<n>We show that QASemConsistency yields factual consistency scores that correlate well with human judgments.
  arXiv  Detail & Related papers  (2024-10-09T22:53:48Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)
• Flames: Benchmarking Value Alignment of LLMs in Chinese [86.73527292670308]
  This paper proposes a value alignment benchmark named Flames. It encompasses both common harmlessness principles and a unique morality dimension that integrates specific Chinese values. Our findings indicate that all the evaluated LLMs demonstrate relatively poor performance on Flames.
  arXiv  Detail & Related papers  (2023-11-12T17:18:21Z)
• Towards Improving Faithfulness in Abstractive Summarization [37.19777407790153]
  We propose a Faithfulness Enhanced Summarization model (FES) to improve fidelity in abstractive summarization. Our model outperforms strong baselines in experiments on CNN/DM and XSum.
  arXiv  Detail & Related papers  (2022-10-04T19:52:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.