Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalization

• URL: http://arxiv.org/abs/2602.15028v1
• Date: Mon, 16 Feb 2026 18:59:42 GMT
• Title: Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalization
• Abstract summary: Large language models (LLMs) are increasingly deployed in privacy-critical and personalization-oriented scenarios.<n>We introduce a large-scale benchmark, PAPerBench, to systematically study how increasing context length influences both personalization quality and privacy protection.<n>The benchmark comprises approximately 29,000 instances with context lengths ranging from 1K to 256K tokens, yielding a total of 377K evaluation questions.
• Score: 4.802305157491253
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly deployed in privacy-critical and personalization-oriented scenarios, yet the role of context length in shaping privacy leakage and personalization effectiveness remains largely unexplored. We introduce a large-scale benchmark, PAPerBench, to systematically study how increasing context length influences both personalization quality and privacy protection in LLMs. The benchmark comprises approximately 29,000 instances with context lengths ranging from 1K to 256K tokens, yielding a total of 377K evaluation questions. It jointly evaluates personalization performance and privacy risks across diverse scenarios, enabling controlled analysis of long-context model behavior. Extensive evaluations across state-of-the-art LLMs reveal consistent performance degradation in both personalization and privacy as context length increases. We further provide a theoretical analysis of attention dilution under context scaling, explaining this behavior as an inherent limitation of soft attention in fixed-capacity Transformers. The empirical and theoretical findings together suggest a general scaling gap in current models -- long context, less focus. We release the benchmark to support reproducible evaluation and future research on scalable privacy and personalization. Code and data are available at https://github.com/SafeRL-Lab/PAPerBench

Related papers

• LUNAR: Benchmarking Personalized Large Language Models on UNiversal User BehAvioR Logs [10.508049806088335]
  Existing personalized LLM benchmarks primarily rely on textual personas or isolated behavioral signals.<n>We introduce LUNAR, the first benchmark for evaluating how LLMs personalize responses from longitudinal app interaction histories across universal daily-life domains.
  arXiv  Detail & Related papers  (2026-08-05T15:11:06Z)
• Differential Privacy in Generative AI Agents: Analysis and Optimal Tradeoffs [22.661656301757663]
  Large language models (LLMs) and AI agents are increasingly integrated into enterprise systems to access internal databases and generate context-aware responses.<n>While such integration improves productivity and decision support, the model outputs may inadvertently reveal sensitive information.<n>This paper develops a probabilistic framework for analyzing privacy leakage in AI agents based on differential privacy.
  arXiv  Detail & Related papers  (2026-03-18T16:35:12Z)
• Private PoEtry: Private In-Context Learning via Product of Experts [58.496468062236225]
  In-context learning (ICL) enables Large Language Models to adapt to new tasks with only a small set of examples at inference time.<n>Existing differential privacy approaches to ICL are either computationally expensive or rely on oversampling, synthetic data generation, or unnecessary thresholding.<n>We reformulate private ICL through the lens of a Product-of-Experts model. This gives a theoretically grounded framework, and the algorithm can be trivially parallelized.<n>We find that our method improves accuracy by more than 30 percentage points on average compared to prior DP-ICL methods, while maintaining strong privacy guarantees.
  arXiv  Detail & Related papers  (2026-02-04T19:56:24Z)
• MultiPriv: Benchmarking Individual-Level Privacy Reasoning in Vision-Language Models [14.942122955210436]
  Modern Vision-Language Models (VLMs) demonstrate sophisticated reasoning, escalating privacy risks.<n>Current privacy benchmarks are structurally insufficient for this new threat.<n>We propose textbfMultiPriv, the first benchmark designed to systematically evaluate individual-level privacy reasoning.
  arXiv  Detail & Related papers  (2025-11-21T04:33:11Z)
• Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmark [10.645183619391135]
  We introduce EAPrivacy, a comprehensive evaluation benchmark designed to quantify the physical-world privacy awareness of embodied agents.<n>EAPrivacy utilizes procedurally generated scenarios across four tiers to test an agent's ability to handle sensitive objects, adapt to changing environments, balance task execution with privacy constraints, and resolve conflicts with social norms.
  arXiv  Detail & Related papers  (2025-09-27T23:39:56Z)
• Urania: Differentially Private Insights into AI Use [102.27238986985698]
  $Urania$ provides end-to-end privacy protection by leveraging DP tools such as clustering, partition selection, and histogram-based summarization.<n>Results show the framework's ability to extract meaningful conversational insights while maintaining stringent user privacy.
  arXiv  Detail & Related papers  (2025-06-05T07:00:31Z)
• A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations [112.81207927088117]
  PersonaConvBench is a benchmark for evaluating personalized reasoning and generation in multi-turn conversations with large language models (LLMs)<n>We benchmark several commercial and open-source LLMs under a unified prompting setup and observe that incorporating personalized history yields substantial performance improvements.
  arXiv  Detail & Related papers  (2025-05-20T09:13:22Z)
• The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGD [46.71175773861434]
  We propose a simple privacy analysis of noisy clipped gradient descent (DP-SGD)<n>We show experimentally that our is predictive of the outcome of privacy auditing applied to various training procedures.<n>We also empirically support our and show existing privacy auditing attacks are bounded by our analysis in both vision and language tasks.
  arXiv  Detail & Related papers  (2024-10-08T16:51:10Z)
• Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networks [72.51255282371805]
  We prove a privacy bound for the KL divergence between model distributions on worst-case neighboring datasets. We find that this KL privacy bound is largely determined by the expected squared gradient norm relative to model parameters during training.
  arXiv  Detail & Related papers  (2023-10-31T16:13:22Z)
• Privacy Preserving Large Language Models: ChatGPT Case Study Based Vision and Framework [6.828884629694705]
  This article proposes the conceptual model called PrivChatGPT, a privacy-generative model for LLMs. PrivChatGPT consists of two main components i.e., preserving user privacy during the data curation/pre-processing together with preserving private context and the private training process for large-scale data.
  arXiv  Detail & Related papers  (2023-10-19T06:55:13Z)
• PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learners [81.571305826793]
  We introduce Contextual Privacy Protection Language Models (PrivacyMind) Our work offers a theoretical analysis for model design and benchmarks various techniques. In particular, instruction tuning with both positive and negative examples stands out as a promising method.
  arXiv  Detail & Related papers  (2023-10-03T22:37:01Z)
• On the Statistical Complexity of Estimation and Testing under Privacy Constraints [17.04261371990489]
  We show how to characterize the power of a statistical test under differential privacy in a plug-and-play fashion. We show that maintaining privacy results in a noticeable reduction in performance only when the level of privacy protection is very high. Finally, we demonstrate that the DP-SGLD algorithm, a private convex solver, can be employed for maximum likelihood estimation with a high degree of confidence.
  arXiv  Detail & Related papers  (2022-10-05T12:55:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.