LLM-Redactor: An Empirical Evaluation of Eight Techniques for Privacy-Preserving LLM Requests

• URL: http://arxiv.org/abs/2604.12064v1
• Date: Mon, 13 Apr 2026 21:05:42 GMT
• Title: LLM-Redactor: An Empirical Evaluation of Eight Techniques for Privacy-Preserving LLM Requests
• Abstract summary: Coding agents and LLM-powered applications routinely send potentially sensitive content to cloud LLM APIs where it may be logged, retained, used for training, or subpoenaed.<n>We present a systematic empirical evaluation of eight techniques for privacy-preserving LLM requests.<n>We implement all eight (or a tractable research-stage subset where deployment is not yet feasible) in an open-source shim compatible with MCP and any OpenAI-compatible API.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Coding agents and LLM-powered applications routinely send potentially sensitive content to cloud LLM APIs where it may be logged, retained, used for training, or subpoenaed. Existing privacy tooling focuses on network-level encryption and organization-level DLP, neither of which addresses the content of prompts themselves. We present a systematic empirical evaluation of eight techniques for privacy-preserving LLM requests: (A) local-only inference, (B) redaction with placeholder restoration, (C) semantic rephrasing, (D) Trusted Execution Environment hosted inference, (E) split inference, (F) fully homomorphic encryption, (G) secret sharing via multi-party computation, and (H) differential-privacy noise. We implement all eight (or a tractable research-stage subset where deployment is not yet feasible) in an open-source shim compatible with MCP and any OpenAI-compatible API. We evaluate the four practical options (A, B, C, H) and their combinations across four workload classes using a ground-truth-labelled leak benchmark of 1,300 samples with 4,014 annotations. Our headline finding is that no single technique dominates: the combination A+B+C (route locally when possible, redact and rephrase the rest) achieves 0.6% combined leak on PII and 31.3% on proprietary code, with zero exact leaks on PII across 500 samples. We present a decision rule that selects the appropriate option(s) from a threat-model budget and workload characterisation. Code, benchmarks, and evaluation harness are released at https://github.com/jayluxferro/llm-redactor.

Related papers

• Privacy-Preserving Topology-Guided Safety for LLM-Based Multi-Agent Systems via Federated Graph Learning [67.04448659688579]
  FGLGuard cuts AgentDojo's ground-success rate by 43% at near-unguarded utility, zero API cost, and negligible capability loss.<n>Federation is not optional: off-the-shelf transfer collapses under distribution shift.<n>Live FGLGuard cuts AgentDojo's ground-success rate by 43% at near-unguarded utility, zero API cost, and negligible capability loss.
  arXiv  Detail & Related papers  (2026-09-02T07:57:12Z)
• Speculative Probing: LLM Monitoring at Speculative-Decoding Cost [8.480625921580154]
  Real-time classification during language model inference is valuable for safety filtering, behavioral analysis, and model monitoring.<n>Current approaches force a trade-off between accuracy and efficiency.<n> speculative-decoding module in recent LLMs can be repurposed for efficient high-quality classification.
  arXiv  Detail & Related papers  (2026-08-28T09:07:58Z)
• LMSM: LLM Security Framework Inspired by Linux Security Modules [56.93644694627783]
  Large language models (LLMs) are increasingly deployed with layered defenses, yet malicious prompts can still bypass them.<n>We present Language Model Security Modules (LMSM), a security framework that adapts the separation behind Linux Security Modules (LSM) to LLM serving.
  arXiv  Detail & Related papers  (2026-08-26T12:13:05Z)
• Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMs [61.80513991207956]
  We propose Anchored Privacy Drifting (APD) to tackle privacy challenges in MLLMs.<n>APD drifts privacy-sensitive elements toward semantically equivalent alternatives while anchoring contextual cues to the source image.<n>We show that our method achieves balanced improvements in both privacy sanitization and content retention.
  arXiv  Detail & Related papers  (2026-06-05T11:41:39Z)
• Willing but Unable: Separating Refusal from Capability in Code LLMs via Abliteration [3.9458538778648964]
  This paper is a preliminary study of abliteration, a low-rank weight edit that replicates out the refusal stream as a tool to remove this barrier.<n> Vulnerability verdicts are produced by a three-tool detector ensemble (QLCode, Sem, Bandit) followed by manual adjudication.
  arXiv  Detail & Related papers  (2026-06-03T20:06:52Z)
• From Attack Simulation to SIEM Rule: Deterministic Detection-as-Code Synthesis with Probe-Level Traceability [51.56484100374058]
  Security teams simulate attacks against their own systems to check whether their monitoring would catch a real intruder.<n>A human bridges that gap by hand, reading each finding and writing the corresponding Sigma rule.<n>We show this translation can be partially automated when probes are drawn from a locked corpus.
  arXiv  Detail & Related papers  (2026-06-03T14:26:25Z)
• CauTion: Knowing When to Trust LLMs for Ensemble Causal Discovery [51.07538881798502]
  Large language models (LLMs) offer a promising source of domain knowledge to complement statistical inference.<n>We propose CauTion, a framework that reliably integrates LLM domain knowledge into an ensemble of statistical causal discovery algorithms.<n>CauTion consistently outperforms both data-centric and LLM-augmented baselines.
  arXiv  Detail & Related papers  (2026-06-02T13:07:43Z)
• Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models [20.61766907174782]
  We propose Dynamic Boundary Evaluation (DBE), which actively locates each model's boundary and places it on a globally comparable difficulty scale.<n>DBE delivers three artifacts: (i) a calibrated item bank covering safety, capability, and truthfulness, with per-item difficulty labels validated across $9$ reference LLMs; (ii) Skill-Guided Boundary Search (SGBS), a search algorithm that finds boundary items for a given target LLM using only API-level query access; and (iii) an evaluation protocol that places a new LLM on a unified ability scale and grows the evaluation set adaptively when the target falls outside
  arXiv  Detail & Related papers  (2026-05-07T13:15:31Z)
• SkillSieve: A Hierarchical Triage Framework for Detecting Malicious AI Agent Skills [0.0]
  SkillSieve is a three-layer detection framework that applies progressively deeper analysis only where needed.<n>On a 400-skill labeled benchmark, SkillSieve achieves 0.800 F1, outperforming ClawVet's 0.421, at an average cost of 0.006 per skill.
  arXiv  Detail & Related papers  (2026-04-08T00:58:48Z)
• A Reliability Evaluation of Hybrid Deterministic-LLM Based Approaches for Academic Course Registration PDF Information Extraction [0.0]
  This study evaluates the reliability of information extraction approaches from KRS documents using three strategies.<n>Experiments were conducted on 140 documents for the LLM based test and 860 documents for the Camelot based pipeline evaluation.
  arXiv  Detail & Related papers  (2026-03-09T02:09:02Z)
• LLM-Guided Probabilistic Fusion for Label-Efficient Document Layout Analysis [6.908972852063454]
  Document layout understanding remains data-intensive despite advances in semi-supervised learning.<n>We present a framework that enhances semi-supervised detection by fusing visual predictions with structural priors.<n>Our method demonstrates consistent gains across model scales.
  arXiv  Detail & Related papers  (2025-11-12T02:25:58Z)
• I Know What You Said: Unveiling Hardware Cache Side-Channels in Local Large Language Model Inference [19.466754645346175]
  Large Language Models (LLMs) that can be deployed locally have recently gained popularity for privacy-sensitive tasks.<n>We unveil novel side-channel vulnerabilities in local LLM inference, which can expose both the victim's input and output text.<n>We design a novel eavesdropping attack framework targeting both open-source and proprietary LLM inference systems.
  arXiv  Detail & Related papers  (2025-05-10T19:06:37Z)
• Towards Copyright Protection for Knowledge Bases of Retrieval-augmented Language Models via Reasoning [58.57194301645823]
  Large language models (LLMs) are increasingly integrated into real-world personalized applications.<n>The valuable and often proprietary nature of the knowledge bases used in RAG introduces the risk of unauthorized usage by adversaries.<n>Existing methods that can be generalized as watermarking techniques to protect these knowledge bases typically involve poisoning or backdoor attacks.<n>We propose name for harmless' copyright protection of knowledge bases.
  arXiv  Detail & Related papers  (2025-02-10T09:15:56Z)
• Reinforcement Learning for Long-Horizon Interactive LLM Agents [56.9860859585028]
  Interactive digital agents (IDAs) leverage APIs of stateful digital environments to perform tasks in response to user requests.<n>We present a reinforcement learning (RL) approach that trains IDAs directly in their target environments.<n>We derive LOOP, a data- and memory-efficient variant of proximal policy optimization.
  arXiv  Detail & Related papers  (2025-02-03T18:35:42Z)
• MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark [57.999567012489706]
  We propose a contamination-free and more challenging benchmark called MMLU-CF.<n>This benchmark reassesses LLMs' understanding of world knowledge by averting both unintentional and malicious data leakage.<n>Our evaluation of mainstream LLMs reveals that the powerful GPT-4o achieves merely a 5-shot score of 73.4% and a 0-shot score of 71.9% on the test set.
  arXiv  Detail & Related papers  (2024-12-19T18:58:04Z)
• SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal [64.9938658716425]
  SORRY-Bench is a proposed benchmark for evaluating large language models' (LLMs) ability to recognize and reject unsafe user requests.<n>First, existing methods often use coarse-grained taxonomy of unsafe topics, and are over-representing some fine-grained topics.<n>Second, linguistic characteristics and formatting of prompts are often overlooked, like different languages, dialects, and more -- which are only implicitly considered in many evaluations.
  arXiv  Detail & Related papers  (2024-06-20T17:56:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.