How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study

• URL: http://arxiv.org/abs/2605.05340v2
• Date: Fri, 08 May 2026 01:54:08 GMT
• Title: How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study
• Abstract summary: Vision-Language Models (VLMs) are increasingly deployed as autonomous cognitive cores for embodied assistants.<n>We present ImmersedPrivacy, an interactive audio-visual evaluation framework that simulates realistic physical environments.<n>Our evaluation of 12 state-of-the-art models reveals consistent deficits.
• Score: 7.537653216205245
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Vision-Language Models (VLMs) are increasingly deployed as autonomous cognitive cores for embodied assistants, evaluating their privacy awareness in physical environments becomes critical. Unlike digital chatbots, these agents operate in intimate spaces, such as homes and hospitals, where they possess the physical agency to observe and manipulate privacy-sensitive information and artifacts. However, current benchmarks remain limited to unimodal, text-based representations that cannot capture the demands of real-world settings. To bridge this gap, we present ImmersedPrivacy, an interactive audio-visual evaluation framework that simulates realistic physical environments using a Unity-based simulator. ImmersedPrivacy evaluates physically grounded privacy awareness across three progressive tiers that test a model's ability to identify sensitive items in cluttered scenes, adapt to shifting social contexts, and resolve conflicts between explicit commands and inferred privacy constraints. Our evaluation of 12 state-of-the-art models reveals consistent deficits. In cluttered scenes, all models exhibit monotonic performance decay as scene complexity grows due to perceptual deficit. When social context shifts, no model exceed 65% selection accuracy. Under conflicting commands, the best model gemini-3.1-pro perfectly balances task completion and privacy preservation in only 51% of cases. These findings reveal that current VLMs in the physical world suffer from perceptual fragility and fail to let their knowledge of privacy cues govern their situated behavior. Our code and data is available at https://github.com/immersed-privacy/immersed-privacy .

Related papers

• VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models [25.266028200777317]
  Speech Language Models (SLMs) are expected to distinguish between users to manage information flow appropriately.<n>Current SLM benchmarks test dialogue ability but overlook speaker identity.<n>We introduce VoxPrivacy, the first benchmark designed to evaluate interactional privacy in SLMs.
  arXiv  Detail & Related papers  (2026-01-27T06:22:14Z)
• Privacy Collapse: Benign Fine-Tuning Can Break Contextual Privacy in Language Models [47.866853046761044]
  We find that diverse, subtle patterns in training data can degrade contextual privacy.<n>Fine-tuned models lose their ability to reason about contextual privacy norms.<n>Our results reveal a critical gap in current safety evaluations.
  arXiv  Detail & Related papers  (2026-01-21T17:53:06Z)
• Single-Pixel Vision-Language Model for Intrinsic Privacy-Preserving Behavioral Intelligence [55.512671026669516]
  We propose the Single-Pixel Vision-Language Model (SP-VLM), a novel framework that reimagines secure environmental monitoring.<n>It achieves intrinsic privacy-by-design by capturing human dynamics through inherently low-dimensional single-pixel modalities.<n>We show that SP-VLM can nonetheless extract meaningful behavioral semantics, enabling robust anomaly detection, people counting, and activity understanding.
  arXiv  Detail & Related papers  (2026-01-21T09:11:26Z)
• Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmark [10.645183619391135]
  We introduce EAPrivacy, a comprehensive evaluation benchmark designed to quantify the physical-world privacy awareness of embodied agents.<n>EAPrivacy utilizes procedurally generated scenarios across four tiers to test an agent's ability to handle sensitive objects, adapt to changing environments, balance task execution with privacy constraints, and resolve conflicts with social norms.
  arXiv  Detail & Related papers  (2025-09-27T23:39:56Z)
• FED-PsyAU: Privacy-Preserving Micro-Expression Recognition via Psychological AU Coordination and Dynamic Facial Motion Modeling [14.30453182275959]
  Micro-expressions (MEs) are brief, low-intensity, often localized facial expressions.<n>They could reveal genuine emotions individuals may attempt to conceal, valuable in contexts like criminal interrogation and psychological counseling.<n>ME recognition (MER) faces challenges, such as small sample sizes and subtle features, which hinder efficient modeling.
  arXiv  Detail & Related papers  (2025-07-28T06:42:15Z)
• Real-Time Privacy Preservation for Robot Visual Perception [21.05457265285284]
  Existing approaches for preserving privacy rely on deep learning models, differential privacy, or cryptography.<n>We develop a method for privacy-constrained video streaming, PCVS, that conceals sensitive objects within real-time video streams.<n>We deploy PCVS on robots in real-time operation and show that the robots operate normally without being compromised.
  arXiv  Detail & Related papers  (2025-05-08T03:27:12Z)
• Activity Recognition on Avatar-Anonymized Datasets with Masked Differential Privacy [64.32494202656801]
  Privacy-preserving computer vision is an important emerging problem in machine learning and artificial intelligence.<n>We present anonymization pipeline that replaces sensitive human subjects in video datasets with synthetic avatars within context.<n>We also proposeMaskDP to protect non-anonymized but privacy sensitive background information.
  arXiv  Detail & Related papers  (2024-10-22T15:22:53Z)
• PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action [54.11479432110771]
  PrivacyLens is a novel framework designed to extend privacy-sensitive seeds into expressive vignettes and further into agent trajectories.<n>We instantiate PrivacyLens with a collection of privacy norms grounded in privacy literature and crowdsourced seeds.<n>State-of-the-art LMs, like GPT-4 and Llama-3-70B, leak sensitive information in 25.68% and 38.69% of cases, even when prompted with privacy-enhancing instructions.
  arXiv  Detail & Related papers  (2024-08-29T17:58:38Z)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory [82.7042006247124]
  We show that even the most capable AI models reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.
  arXiv  Detail & Related papers  (2023-10-27T04:15:30Z)
• Robustness Threats of Differential Privacy [70.818129585404]
  We experimentally demonstrate that networks, trained with differential privacy, in some settings might be even more vulnerable in comparison to non-private versions. We study how the main ingredients of differentially private neural networks training, such as gradient clipping and noise addition, affect the robustness of the model.
  arXiv  Detail & Related papers  (2020-12-14T18:59:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.