Privacy-Preserving Data Drift Detection and Recovery for Large-Scale LLM Applications via Proxy Representations

• URL: http://arxiv.org/abs/2608.08245v1
• Date: Sat, 08 Aug 2026 17:15:18 GMT
• Title: Privacy-Preserving Data Drift Detection and Recovery for Large-Scale LLM Applications via Proxy Representations
• Abstract summary: ProxyDrift is a framework that identifies and measures drift between production traffic and offline evaluation sets.<n>Our approach operates entirely on non-PII proxy representations.
• Score: 5.534194634878958
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM applications deployed at scale face a fundamental challenge: privacy constraints prevent direct inspection of user interactions, making it difficult to obtain any representative evaluation dataset or to track the ongoing evolution of production traffic. We present ProxyDrift, a framework that (i) identifies and measures drift between production traffic and offline evaluation sets, and (ii) constructs and refreshes those evaluation sets accordingly; all without access to raw user data. Our approach operates entirely on non-PII proxy representations: structured, multi-dimensional descriptors derived from LLM-based classification of user interactions. We introduce (1) a chance-calibrated, redundancy-aware (RA) alignment score that aggregates per-dimension drift measurements via mutual information; (2) a conditional sampler that generates synthetic proxies respecting inter-dimensional dependencies; (3) a roundtrip consistency analysis that exposes generator/classifier disagreements and guides proxy taxonomy refinement; and (4) a feedback-linkage analysis that ties per-dimension and per-value proxy distributions to user satisfaction, surfacing actionable failure and success modes. Serving hundreds of millions of users, ProxyDrift enables continuous drift monitoring and targeted synthetic data generation without exposing sensitive user data. Experiments confirm strong roundtrip consistency, discriminator-level indistinguishability of synthetic queries from human queries, and tight end-to-end alignment (RA~0.9) with production.

Related papers

• CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement [50.91045324738942]
  CLAIM is an uncertainty-driven framework for active clarification learning in open-domain settings.<n>We propose an entropy-driven synthetic data generation pipeline that integrates uncertainty estimation with semantic clustering and reasoning-based judgments.<n> Experimental results demonstrate that CLAIM can learn stable and generalizable clarification strategies without relying on manually labeled data.
  arXiv  Detail & Related papers  (2026-08-12T04:27:45Z)
• Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation [2.733105115068754]
  We introduce a scalable, inference-only data valuation pipeline that approximates the Shapley value without iterative model retraining.<n>We demonstrate the pipeline's efficacy in sanitizing two heavily vetted alignment datasets.
  arXiv  Detail & Related papers  (2026-07-24T03:31:44Z)
• Phantoms and Disclosures: a Causal Framework for Auditing Synthetic Data [56.86147283213652]
  We present a customizable empirical auditing framework designed to detect and explain data disclosures.<n>Our framework introduces a mechanism to distinguish between "true disclosures"-where the system directly reproduces a user's information-and "phantom disclosures"-where the system incidentally generates a user's data.
  arXiv  Detail & Related papers  (2026-06-15T16:54:02Z)
• Decoupling Identity from Utility: Privacy-by-Design Frameworks for Financial Ecosystems [2.8555760710139393]
  This paper explores Differentially Private (DP) synthetic data as a robust "Privacy by Design" framework to resolve this conflict.<n>We examine two distinct generative paradigms: Direct Tabular Synthesis, which reconstructs high-fidelity joint distributions from raw data, and DP-Seeded Agent-Based Modeling (ABM), which uses DP-protected aggregates to parameterize complex, stateful simulations.
  arXiv  Detail & Related papers  (2026-04-16T00:07:32Z)
• An Agentic Workflow for Detecting Personally Identifiable Information in Crash Narratives [25.2699036413582]
  PII appears sparsely and inconsistently in crash narratives.<n>Existing rule-based approaches often fail to capture context-dependent PII.<n>This study develops and evaluates a locally deployable, agentic workflow for PII detection in crash narratives.
  arXiv  Detail & Related papers  (2026-04-15T05:03:20Z)
• Towards Generalizable Deepfake Detection via Real Distribution Bias Correction [62.89674593681203]
  We introduce the Real Distribution Bias Correction (RDBC) framework, which consists of two key components.<n>RDBC achieves state-of-the-art performance in both in-domain and cross-domain deepfake detection.
  arXiv  Detail & Related papers  (2026-03-14T16:11:00Z)
• Knowledge-to-Data: LLM-Driven Synthesis of Structured Network Traffic for Testbed-Free IDS Evaluation [0.4893345190925178]
  This paper investigates whether Large Language Models (LLMs) can operate as controlled knowledge-to-data engines for generating structured synthetic network traffic datasets.<n>We propose a methodology that combines protocol documentation, attack semantics, and explicit statistical rules to condition LLMs without fine-tuning or access to raw samples.<n>Results show that, under explicit constraints, LLM-generated datasets can closely approximate the statistical and structural characteristics of real network traffic.
  arXiv  Detail & Related papers  (2026-01-08T15:31:33Z)
• Agent4FaceForgery: Multi-Agent LLM Framework for Realistic Face Forgery Detection [108.5042835056188]
  This work introduces Agent4FaceForgery to address two fundamental problems.<n>How to capture the diverse intents and iterative processes of human forgery creation.<n>How to model the complex, often adversarial, text-image interactions that accompany forgeries in social media.
  arXiv  Detail & Related papers  (2025-09-16T01:05:01Z)
• Byzantine-Robust Federated Learning Using Generative Adversarial Networks [1.4091801425319963]
  Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, but its robustness is threatened by Byzantine behaviors such as data and model poisoning.<n>We present a defense framework that addresses these challenges by leveraging a conditional generative adversarial network (cGAN) at the server to synthesize representative data for validating client updates.<n>This approach eliminates reliance on external datasets, adapts to diverse attack strategies, and integrates seamlessly into standard FL.
  arXiv  Detail & Related papers  (2025-03-26T18:00:56Z)
• Robust Utility-Preserving Text Anonymization Based on Large Language Models [80.5266278002083]
  Anonymizing text that contains sensitive information is crucial for a wide range of applications.<n>Existing techniques face the emerging challenges of the re-identification ability of large language models.<n>We propose a framework composed of three key components: a privacy evaluator, a utility evaluator, and an optimization component.
  arXiv  Detail & Related papers  (2024-07-16T14:28:56Z)
• Geometry-Aware Instrumental Variable Regression [56.16884466478886]
  We propose a transport-based IV estimator that takes into account the geometry of the data manifold through data-derivative information. We provide a simple plug-and-play implementation of our method that performs on par with related estimators in standard settings.
  arXiv  Detail & Related papers  (2024-05-19T17:49:33Z)
• Debiasing Recommendation by Learning Identifiable Latent Confounders [49.16119112336605]
  Confounding bias arises due to the presence of unmeasured variables that can affect both a user's exposure and feedback. Existing methods either (1) make untenable assumptions about these unmeasured variables or (2) directly infer latent confounders from users' exposure. We propose a novel method, i.e., identifiable deconfounder (iDCF), which leverages a set of proxy variables to resolve the aforementioned non-identification issue.
  arXiv  Detail & Related papers  (2023-02-10T05:10:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.