LLM-CEG: Extending the Classification Error Gauge Framework for Privacy Auditing of Large Language Models

• URL: http://arxiv.org/abs/2604.23795v1
• Date: Sun, 26 Apr 2026 16:39:37 GMT
• Title: LLM-CEG: Extending the Classification Error Gauge Framework for Privacy Auditing of Large Language Models
• Abstract summary: This paper extends the Classification Error Gauge (x-CEG) framework to privacy auditing of Large Language Models (LLMs)<n>We propose a systematic framework that employs membership inference attack (MIA) success rates as an empirical privacy gauge and model perplexity as a utility gauge.<n>A proof-of-concept prototype fine-tunes DistilGPT-2 on a synthetic clinical PII dataset under four privacy regimes using DP-SGD.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper extends the Classification Error Gauge (x-CEG) framework, originally developed for measuring the privacy-utility trade-off in tabular datasets, to privacy auditing of Large Language Models (LLMs). We propose LLM-CEG, a systematic framework that employs membership inference attack (MIA) success rates as an empirical privacy gauge and model perplexity as a utility gauge, iteratively adjusting differential privacy parameters until both thresholds are jointly satisfied. A proof-of-concept prototype fine-tunes DistilGPT-2 on a synthetic clinical PII dataset under four privacy regimes using DP-SGD. Results indicate that DP-SGD reduces MIA attacker advantage by 71.5% while simultaneously improving out-of-distribution utility by 47-50% relative to the overfitted baseline, suggesting that differential privacy may act as implicit regularization under narrow fine-tuning conditions. We further extend the SIED engineering framework to the LLM context as LLM-SIED, providing an auditable, regulator-aligned process for privacy-compliant LLM deployment.

Related papers

• Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models [35.164227002323706]
  We investigate privacy risks under DP adaptations in large language models (LLMs)<n>We benchmark these risks using state-of-the-art attacks such as robust membership inference and canary data extraction.<n>We find that parameter-efficient fine-tuning methods, such as LoRA, achieve the highest empirical privacy protection for OOD data.
  arXiv  Detail & Related papers  (2026-06-08T12:21:02Z)
• Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMs [61.80513991207956]
  We propose Anchored Privacy Drifting (APD) to tackle privacy challenges in MLLMs.<n>APD drifts privacy-sensitive elements toward semantically equivalent alternatives while anchoring contextual cues to the source image.<n>We show that our method achieves balanced improvements in both privacy sanitization and content retention.
  arXiv  Detail & Related papers  (2026-06-05T11:41:39Z)
• Ensemble Privacy Defense for Knowledge-Intensive LLMs against Membership Inference Attacks [21.852575873751917]
  Inference Attacks pose serious threats to privacy and trust in sensitive domains.<n>We introduce a novel, model-agnostic defense framework, Ensemble Privacy Defense (EPD)<n>EPD reduces MIA success by up to 27.8% for SFT and 526.3% for RAG compared to inference-time baseline.
  arXiv  Detail & Related papers  (2025-12-01T18:12:18Z)
• AVEC: Bootstrapping Privacy for Local LLMs [0.0]
  AVEC is a framework for bootstrapping privacy for local language models.<n>It enforces privacy at the edge with explicit verifiability for delegated queries.
  arXiv  Detail & Related papers  (2025-09-10T07:59:01Z)
• Federated Learning-Enabled Hybrid Language Models for Communication-Efficient Token Transmission [87.68447072141402]
  Hybrid Language Models (HLMs) combine the low-latency efficiency of Small Language Models (SLMs) on edge devices with the high accuracy of Large Language Models (LLMs) on centralized servers.<n>We propose FedHLM, a communication-efficient HLM framework that integrates uncertainty-aware inference with Federated Learning (FL)
  arXiv  Detail & Related papers  (2025-06-30T02:56:11Z)
• SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attacks [17.77094760401298]
  We study the vulnerability of fine-tuned large language models to membership inference attacks (MIAs)<n>We propose SOFT, a novel defense technique that mitigates privacy leakage by leveraging influential data selection with an adjustable parameter to balance utility preservation and privacy protection.
  arXiv  Detail & Related papers  (2025-06-12T07:23:56Z)
• LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization [59.75242204923353]
  We introduce LLM-Lasso, a framework that leverages large language models (LLMs) to guide feature selection in Lasso regression.<n>LLMs generate penalty factors for each feature, which are converted into weights for the Lasso penalty using a simple, tunable model.<n>Features identified as more relevant by the LLM receive lower penalties, increasing their likelihood of being retained in the final model.
  arXiv  Detail & Related papers  (2025-02-15T02:55:22Z)
• Differentially Private Steering for Large Language Model Alignment [55.30573701583768]
  We present the first study of aligning Large Language Models with private datasets.<n>Our work proposes the Private Steering for LLM Alignment (PSA) algorithm to edit activations with differential privacy guarantees.<n>Our results show that PSA achieves DP guarantees for LLM alignment with minimal loss in performance.
  arXiv  Detail & Related papers  (2025-01-30T17:58:36Z)
• Communication-Efficient and Privacy-Adaptable Mechanism for Federated Learning [54.20871516148981]
  We introduce the Communication-Efficient and Privacy-Adaptable Mechanism (CEPAM)<n>CEPAM achieves communication efficiency and privacy protection simultaneously.<n>We theoretically analyze the privacy guarantee of CEPAM and investigate the trade-offs among user privacy and accuracy of CEPAM.
  arXiv  Detail & Related papers  (2025-01-21T11:16:05Z)
• HAFLQ: Heterogeneous Adaptive Federated LoRA Fine-tuned LLM with Quantization [55.972018549438964]
  Federated fine-tuning of pre-trained Large Language Models (LLMs) enables task-specific adaptation across diverse datasets while preserving privacy.<n>We propose HAFLQ (Heterogeneous Adaptive Federated Low-Rank Adaptation Fine-tuned LLM with Quantization), a novel framework for efficient and scalable fine-tuning of LLMs in heterogeneous environments.<n> Experimental results on the text classification task demonstrate that HAFLQ reduces memory usage by 31%, lowers communication cost by 49%, improves accuracy by 50%, and achieves faster convergence compared to the baseline method.
  arXiv  Detail & Related papers  (2024-11-10T19:59:54Z)
• Ingest-And-Ground: Dispelling Hallucinations from Continually-Pretrained LLMs with RAG [2.7972592976232833]
  We continually pre-train the base LLM model with a privacy-specific knowledge base and then augment it with a semantic RAG layer. Our evaluations demonstrate that this approach enhances the model performance (as much as doubled metrics compared to out-of-box LLM) in handling privacy-related queries.
  arXiv  Detail & Related papers  (2024-09-30T20:32:29Z)
• Noisy Neighbors: Efficient membership inference attacks against LLMs [2.666596421430287]
  This paper introduces an efficient methodology that generates textitnoisy neighbors for a target sample by adding noise in the embedding space. Our findings demonstrate that this approach closely matches the effectiveness of employing shadow models, showing its usability in practical privacy auditing scenarios.
  arXiv  Detail & Related papers  (2024-06-24T12:02:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.