Semantic Leakage from Image Embeddings

• URL: http://arxiv.org/abs/2601.22929v2
• Date: Mon, 02 Feb 2026 09:20:53 GMT
• Title: Semantic Leakage from Image Embeddings
• Abstract summary: We show that semantic leakage does not require exact reconstruction of the original image.<n>We propose a lightweight inference framework that reveals semantic information from standalone compressed image embeddings.
• Score: 33.36179164881091
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Image embeddings are generally assumed to pose limited privacy risk. We challenge this assumption by formalizing semantic leakage as the ability to recover semantic structures from compressed image embeddings. Surprisingly, we show that semantic leakage does not require exact reconstruction of the original image. Preserving local semantic neighborhoods under embedding alignment is sufficient to expose the intrinsic vulnerability of image embeddings. Crucially, this preserved neighborhood structure allows semantic information to propagate through a sequence of lossy mappings. Based on this conjecture, we propose Semantic Leakage from Image Embeddings (SLImE), a lightweight inference framework that reveals semantic information from standalone compressed image embeddings, incorporating a locally trained semantic retriever with off-the-shelf models, without training task-specific decoders. We thoroughly validate each step of the framework empirically, from aligned embeddings to retrieved tags, symbolic representations, and grammatical and coherent descriptions. We evaluate SLImE across a range of open and closed embedding models, including GEMINI, COHERE, NOMIC, and CLIP, and demonstrate consistent recovery of semantic information across diverse inference tasks. Our results reveal a fundamental vulnerability in image embeddings, whereby the preservation of semantic neighborhoods under alignment enables semantic leakage, highlighting challenges for privacy preservation.1

Related papers

• The Illusion of High Utility in Safety Alignment of Text-to-Image Diffusion Models [49.67749928542536]
  Safety alignment of text-to-image (T2I) diffusion models aims to suppress harmful generations while preserving utility on benign prompts.<n>Recent methods often appear to deliver high safety with high utility, but this conclusion rests largely on coarse global utility metrics.<n>We show that when utility is measured with structured evaluation, this illusion breaks: onA (Text-to-Image Faithfulness evaluation with Question Answering)<n>We propose StructureAware Geometric Regularization (SAGE), a safety alignment objective that explicitly preserves embedding spread and inter-prompt relational structure.
  arXiv  Detail & Related papers  (2026-07-01T04:00:27Z)
• DQE-CIR: Distinctive Query Embeddings through Learnable Attribute Weights and Target Relative Negative Sampling in Composed Image Retrieval [53.482391830683014]
  Composed image retrieval (CIR) addresses the task of retrieving a target image by jointly interpreting a reference image and a modification text that specifies the intended change.<n>Most existing methods are still built upon contrastive learning frameworks that treat the ground truth image as the only positive instance and all remaining images as negatives.<n>We propose distinctive query embeddings through learnable attribute weights and target relative negative sampling.
  arXiv  Detail & Related papers  (2026-03-04T13:17:44Z)
• CapRecover: A Cross-Modality Feature Inversion Attack Framework on Vision Language Models [11.201984255831126]
  CapRecover recovers high-level semantic content, such as labels or captions, directly from intermediate features without image reconstruction.<n>We introduce a simple yet effective protection method: adding random noise to intermediate features at each layer and removing the noise in the next layer.
  arXiv  Detail & Related papers  (2025-07-30T16:42:02Z)
• Semantic-Syntactic Discrepancy in Images (SSDI): Learning Meaning and Order of Features from Natural Images [7.148054923510877]
  We propose the concept of "image grammar", comprising "image semantics" and "image syntax"<n>We present a semi-supervised two-stage method for learning the image grammar of visual elements and environments solely from natural images.<n>The efficacy of the proposed approach is then demonstrated by achieving detection rates ranging from 70% to 90% on corruptions generated from CelebA and SUN-RGBD datasets.
  arXiv  Detail & Related papers  (2024-01-31T00:16:02Z)
• EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models [52.3015009878545]
  We develop an image segmentor capable of generating fine-grained segmentation maps without any additional training. Our framework identifies semantic correspondences between image pixels and spatial locations of low-dimensional feature maps. In extensive experiments, the produced segmentation maps are demonstrated to be well delineated and capture detailed parts of the images.
  arXiv  Detail & Related papers  (2024-01-22T07:34:06Z)
• Compressive Sensing with Tensorized Autoencoder [22.89029876274012]
  In many cases, different images in a collection are articulated versions of one another. In this paper, our goal is to recover images without access to the ground-truth (clean) images using the articulations as structural prior to the data. We propose to learn autoencoder with tensor ring factorization on the the embedding space to impose structural constraints on the data.
  arXiv  Detail & Related papers  (2023-03-10T22:59:09Z)
• Exploring Set Similarity for Dense Self-supervised Representation Learning [96.35286140203407]
  We propose to explore textbfset textbfsimilarity (SetSim) for dense self-supervised representation learning. We generalize pixel-wise similarity learning to set-wise one to improve the robustness because sets contain more semantic and structure information. Specifically, by resorting to attentional features of views, we establish corresponding sets, thus filtering out noisy backgrounds that may cause incorrect correspondences.
  arXiv  Detail & Related papers  (2021-07-19T09:38:27Z)
• Context-Aware Image Inpainting with Learned Semantic Priors [100.99543516733341]
  We introduce pretext tasks that are semantically meaningful to estimating the missing contents. We propose a context-aware image inpainting model, which adaptively integrates global semantics and local features.
  arXiv  Detail & Related papers  (2021-06-14T08:09:43Z)
• Semantic Segmentation of Remote Sensing Images with Sparse Annotations [23.22039010678414]
  Training Convolutional Neural Networks (CNNs) for very high resolution images requires a large quantity of high-quality pixel-level annotations. We propose a framework for semantic segmentation of aerial images based on incomplete annotations. We propose the FEature and Spatial relaTional regulArization (FESTA) method to complement the supervised task with an unsupervised learning signal.
  arXiv  Detail & Related papers  (2021-01-10T07:21:21Z)
• Image Inpainting Guided by Coherence Priors of Semantics and Textures [62.92586889409379]
  We introduce coherence priors between the semantics and textures which make it possible to concentrate on completing separate textures in a semantic-wise manner. We also propose two coherence losses to constrain the consistency between the semantics and the inpainted image in terms of the overall structure and detailed textures.
  arXiv  Detail & Related papers  (2020-12-15T02:59:37Z)
• Edge Guided GANs with Contrastive Learning for Semantic Image Synthesis [194.1452124186117]
  We propose a novel ECGAN for the challenging semantic image synthesis task. Our ECGAN achieves significantly better results than state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-03-31T01:23:21Z)
• Progressive Domain-Independent Feature Decomposition Network for Zero-Shot Sketch-Based Image Retrieval [15.955284712628444]
  We propose a Progressive Domain-independent Feature Decomposition (PDFD) network for ZS-SBIR. Specifically, PDFD decomposes visual features into domain features and semantic ones, and then the semantic features are projected into common space as retrieval features for ZS-SBIR.
  arXiv  Detail & Related papers  (2020-03-22T12:07:23Z)
• Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes [54.836331922449666]
  We propose a Semantic Guidance and Evaluation Network (SGE-Net) to update the structural priors and the inpainted image. It utilizes semantic segmentation map as guidance in each scale of inpainting, under which location-dependent inferences are re-evaluated. Experiments on real-world images of mixed scenes demonstrated the superiority of our proposed method over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-03-15T17:49:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.