To Redact, or not to Redact? A Local LLM Approach to Deliberative Process Privilege Classification

• URL: http://arxiv.org/abs/2605.10211v1
• Date: Mon, 11 May 2026 08:55:09 GMT
• Title: To Redact, or not to Redact? A Local LLM Approach to Deliberative Process Privilege Classification
• Abstract summary: We build on prior research to perform automatic sensitivity classification for the FOIA Exemption 5 deliberative process privilege.<n>We perform sensitivity classification with a small, local model, deployable on consumer-grade hardware.<n>We find that sentences that are predicted as deliberative contain more verbs that indicate the expression of opinions.
• Score: 0.2979301924660567
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Government transparency laws, like the Freedom of Information (FOIA) acts in the United States and United Kingdom, and the Woo (Open Government Act) in the Netherlands, grant citizens the right to directly request documents from the government. As these documents might contain sensitive information, such as personal information or threats to national security, the laws allow governments to redact sensitive parts of the documents prior to release. We build on prior research to perform automatic sensitivity classification for the FOIA Exemption 5 deliberative process privilege using Large Language Models (LLMs). However, processing documents not yet cleared for review via third-party cloud APIs is often legally or politically untenable. Therefore, in this work, we perform sensitivity classification with a small, local model, deployable on consumer-grade hardware (Qwen3.5 9B). We compare eight variants of applying LLMs for sentence classification, using well-known prompting techniques, and find that a combination of Chain-of-Thought prompting and few-shot prompting with error-based examples outperforms classification models of earlier work in terms of recall and F2 score. This method also closely approaches the performance of a widely-used, cost-efficient commercial model (Gemini 2.5 Flash). In an additional analysis, we find that sentences that are predicted as deliberative contain more verbs that indicate the expression of opinions, and are more often phrased in in first-person. Above all, deliberativeness seems characterized by the presence of a combination of multiple indicators, in particular the combination of first-person words with a verb for expressing opinion.

Related papers

• How to Recognize New Words: A Comparison Between Context Biasing Methods and Speech LLMs [61.601626186678146]
  We evaluate two context biasing methods based on Whisper against three speech large language models (LLMs)<n>The context biasing methods cut biased WER by up to 88% relative while leaving other words largely unaffected.<n> Speech LLMs excel on read speech but generalize less well to non-read speech, and prove sensitive to distractor count and prompt word order.
  arXiv  Detail & Related papers  (2026-08-06T08:45:33Z)
• FOI-O: An NZ-first ontology and verification methods package for Freedom of Information process modelling [0.0]
  FOI-O is a reusable process-modelling method and verification infrastructure for Freedom of Information administration.<n> FOI-O NZ, based on the New Zealand Official Information Act, is the only implemented and validated jurisdictional profile.
  arXiv  Detail & Related papers  (2026-07-03T04:46:06Z)
• RedactionBench [0.0]
  Large Language Models are increasingly applied to sensitive domains that require redaction of personally identifiable information (PII)<n>RedactionBench is a benchmark comprising 200 diverse documents across 11 domains.<n>R-Score is a novel character-level metric that treats semantically similar redactions equally.
  arXiv  Detail & Related papers  (2026-06-17T07:51:56Z)
• CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval [5.305110876082343]
  CaseFacts is a benchmark for verifying legal claims against U.S. Supreme Court precedents.<n>The dataset consists of 6,294 claims categorized as Supported, Refuted, or Overruled.
  arXiv  Detail & Related papers  (2026-01-23T23:41:46Z)
• AnnoCaseLaw: A Richly-Annotated Dataset For Benchmarking Explainable Legal Judgment Prediction [56.797874973414636]
  AnnoCaseLaw is a first-of-its-kind dataset of 471 meticulously annotated U.S. Appeals Court negligence cases.<n>Our dataset lays the groundwork for more human-aligned, explainable Legal Judgment Prediction models.<n>Results demonstrate that LJP remains a formidable task, with application of legal precedent proving particularly difficult.
  arXiv  Detail & Related papers  (2025-02-28T19:14:48Z)
• Attribution analysis of legal language as used by LLM [0.0]
  We use two publicly-available legal datasets, a simpler binary classification task of overruling'' texts, and a more elaborate multiple choice task identifying holding'' judicial decisions.<n>We find that while all models correctly classify some test examples from the casehold task, other examples can only be identified by only one, model, and attribution can be used to highlight the reasons for this.
  arXiv  Detail & Related papers  (2025-01-28T22:48:29Z)
• From Citations to Criticality: Predicting Legal Decision Influence in the Multilingual Swiss Jurisprudence [16.529070321280447]
  We introduce the Criticality Prediction dataset, a novel resource for evaluating case prioritization.<n>Our dataset features a two-tier labeling system: (1) the binary LD-Label, identifying cases published as Leading Decisions (LD), and (2) the more granular Citation-Label, ranking cases by their citation frequency and recency.<n>We evaluate several multilingual models, including both smaller fine-tuned models and large language models in a zero-shot setting.
  arXiv  Detail & Related papers  (2024-10-17T11:43:16Z)
• DELTA: Pre-train a Discriminative Encoder for Legal Case Retrieval via Structural Word Alignment [55.91429725404988]
  We introduce DELTA, a discriminative model designed for legal case retrieval. We leverage shallow decoders to create information bottlenecks, aiming to enhance the representation ability. Our approach can outperform existing state-of-the-art methods in legal case retrieval.
  arXiv  Detail & Related papers  (2024-03-27T10:40:14Z)
• Fine-tuning Language Models for Factuality [96.5203774943198]
  Large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines. Yet language models are prone to making convincing but factually inaccurate claims, often referred to as 'hallucinations' In this work, we fine-tune language models to be more factual, without human labeling.
  arXiv  Detail & Related papers  (2023-11-14T18:59:15Z)
• Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs [65.9077733300329]
  Large Language Models (LLMs) have the potential to greatly enhance the analysis of public affairs documents. LLMs can be of great use to process domain-specific documents, such as those in the domain of public affairs.
  arXiv  Detail & Related papers  (2023-06-05T13:35:01Z)
• GERE: Generative Evidence Retrieval for Fact Verification [57.78768817972026]
  We propose GERE, the first system that retrieves evidences in a generative fashion. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-04-12T03:49:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.