The Vision Encoder as a Privacy Boundary: Visual-Token Side Channels in Encoder-Free Vision-Language Models

• URL: http://arxiv.org/abs/2606.14783v1
• Date: Wed, 10 Jun 2026 10:39:04 GMT
• Title: The Vision Encoder as a Privacy Boundary: Visual-Token Side Channels in Encoder-Free Vision-Language Models
• Abstract summary: A vision encoder compresses image pixels into semantic embeddings, implicitly acting as a privacy boundary.<n>A vision-free vision-language models (VLMs) remove this boundary by routing image patches directly into the language-model token stream.<n>The attack survives clutter, realistic document degradation and zero-shot transfer to public document images, and it resists value-level defenses such as additive noise and quantization.
• Score: 5.864346468963738
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A vision encoder compresses image pixels into semantic embeddings, implicitly acting as a privacy boundary by preserving semantic content while attenuating pixel-local detail required for exact text recovery. Encoder-free vision-language models (VLMs) remove this boundary by routing image patches directly into the language-model token stream, thereby exposing an architectural privacy attack surface: intermediate visual tokens become a pre-output side channel. Under a token-access adversary, decoders invert visual-token streams from two encoder-free VLMs, Gemma4 and Fuyu, recovering recognizable image structure and readable held-out access codes, whereas matched encoder-based controls localize target regions but recover no exact strings. Within-model ablations show that the operative factor is spatial sampling fidelity of the visual-token grid, especially character-direction sampling density, rather than token or value count. The leakage is not limited to exported tokens: Gemma4 layer-0 key-value cache tensors are directly invertible, placing the side channel within KV caches commonly persisted by production serving stacks for decoding efficiency. The attack survives clutter, realistic document degradation, and zero-shot transfer to public document images, and it resists value-level defenses such as additive noise and quantization. Effective mitigation must therefore reduce spatial sampling, making removal of the vision encoder a first-class privacy decision in VLM deployment.

Related papers

• Compressing AI Traffic: Standardized Neural Network Coding of Visual-Token Representations in Split Vision-Language Inference [15.357950338341285]
  Video-MME accuracy remains close to the uncompressed reference up to a 98% reduction of the transmitted BF16 tensor and only then collapses.<n>Open-ended MLVU generation shows the same plateau-and- robustness profile under an LLM judge.
  arXiv  Detail & Related papers  (2026-09-01T13:07:32Z)
• PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference [15.233755808671871]
  Existing visual token pruning methods exhibit two fundamental limitations.<n>Most approaches operate exclusively post-vision encoder, leaving the substantial latency of the visual encoding phase unoptimized.<n>We propose PACE, a framework that accelerates both the vision encoder and the Large Language Model.
  arXiv  Detail & Related papers  (2026-08-27T14:52:09Z)
• Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding [49.540751364954666]
  Visual Token Codec separates global and patch tokens into dedicated coding paths.<n>We show that VTC consistently outperforms representative ViT feature coding baselines on classification, segmentation, and detection tasks.
  arXiv  Detail & Related papers  (2026-08-09T17:32:04Z)
• CrossFlow: One-Step Generation Across Latent and Pixel Spaces [61.08358796319721]
  CrossFlow is a cross-space flow formulation that maps noisy latent inputs directly to pixel-space images.<n>On class-conditional ImageNet-1k at $256times256$, CrossFlow-XL achieves 1.62 FID with one function evaluation.
  arXiv  Detail & Related papers  (2026-06-18T09:10:28Z)
• FastOCR: Dynamic Visual Fixation via KV Cache Pruning for Efficient Document Parsing [51.905216364362325]
  We propose FastOCR, a training-free framework with two complementary modules.<n>FastOCR retains 98% of the unpruned model's accuracy while attending to only 5% of the visual tokens per decoding step.
  arXiv  Detail & Related papers  (2026-05-17T13:39:47Z)
• BackdoorIDS: Zero-shot Backdoor Detection for Pretrained Vision Encoder [13.008131372229458]
  BackdoorIDS is a zero-shot, inference-time backdoor samples detection method for pretrained vision encoders.<n>It consistently outperforms existing defenses across diverse attack types, datasets, and model families.<n>BackdoorIDS is compatible with a wide range of encoder architectures, including CNNs, ViTs, CLIP, and LLaVA-1.5.
  arXiv  Detail & Related papers  (2026-03-12T08:32:19Z)
• High-Fidelity Text-to-Image Generation from Pre-Trained Vision-Language Models via Distribution-Conditioned Diffusion Decoding [64.13126192228604]
  We propose a diffusion-based decoding framework that enhances image fidelity by training only a diffusion decoder on the output image-token logits.<n>A lightweight Logit aligns training-time proxy logits from the VQ-VAE encoder with VLM-generated logits, mitigating the train-inference gap.<n>Our method consistently improves visual fidelity for both VQ-VAE reconstructions and text-to-image generations from VLM-predicted tokens.
  arXiv  Detail & Related papers  (2026-03-11T07:02:48Z)
• HybridFlow: Infusing Continuity into Masked Codebook for Extreme Low-Bitrate Image Compression [51.04820313355164]
  HyrbidFlow combines the continuous-feature-based and codebook-based streams to achieve both high perceptual quality and high fidelity under extreme lows. Experimental results demonstrate superior performance across several datasets under extremely lows.
  arXiv  Detail & Related papers  (2024-04-20T13:19:08Z)
• Neural Image Compression Using Masked Sparse Visual Representation [17.229601298529825]
  We study neural image compression based on the Sparse Visual Representation (SVR), where images are embedded into a discrete latent space spanned by learned visual codebooks. By sharing codebooks with the decoder, the encoder transfers codeword indices that are efficient and cross-platform robust. We propose a Masked Adaptive Codebook learning (M-AdaCode) method that applies masks to the latent feature subspace to balance and reconstruction quality.
  arXiv  Detail & Related papers  (2023-09-20T21:59:23Z)
• Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots [65.302728042116]
  We investigate the emergence of objects in visual perception in the absence of any semantic annotation.<n>The resulting model has received no supervision, does not use any pre-trained features, and yet it can segment the domain of an image into multiple moving regions.<n>The resulting motion segmentation method can handle an unknown and varying number of objects in real-time.
  arXiv  Detail & Related papers  (2023-04-04T00:26:13Z)
• MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrieval [43.2299969152561]
  Methods for text-to-video retrieval on four datasets with both zero-shot and fine-tune evaluation protocols. Our method outperforms state-of-the-art methods for text-to-video retrieval on four datasets with both zero-shot and fine-tune evaluation protocols.
  arXiv  Detail & Related papers  (2022-04-26T16:06:31Z)
• Modeling Lost Information in Lossy Image Compression [72.69327382643549]
  Lossy image compression is one of the most commonly used operators for digital images. We propose a novel invertible framework called Invertible Lossy Compression (ILC) to largely mitigate the information loss problem.
  arXiv  Detail & Related papers  (2020-06-22T04:04:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.