Invisible Shortcuts: Why Vision Encoders Know Your Camera

• URL: http://arxiv.org/abs/2608.05424v1
• Date: Wed, 05 Aug 2026 21:36:34 GMT
• Title: Invisible Shortcuts: Why Vision Encoders Know Your Camera
• Abstract summary: Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals.<n>We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level.<n>We show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance under metadata distribution shifts.
• Score: 18.18309407263728
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep vision models exploit shortcuts, relying on cues that correlate with supervision signals. Prior work has focused on visible biases, such as object-background or texture correlations. We identify a different source of shortcut learning: invisible metadata traces embedded at the pixel level, for metadata such as image processing and photo acquisition. We hypothesize that large-scale semantic supervision, whether through categorical labels (ImageNet) or billion-scale captions (LAION), naturally induces metadata-semantics correlations during pretraining, leading models to convert low-level signals into predictive features. By introducing controlled metadata-semantics correlations, we show that stronger ones produce systematically higher sensitivity to metadata traces and larger performance degradation under metadata distribution shifts. We further explore mitigation strategies applied during and after pretraining that reduce sensitivity not only to targeted metadata but also to unseen ones, without sacrificing performance on downstream tasks. Metadata sensitivity also has a positive side: it partly explains the strong generated-image detection ability of some encoders, while its mitigation can improve out-of-distribution generalization. Code: https://github.com/ryan-caesar-ramos/visual-encoder-traces

Related papers

• Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints [4.598350938853635]
  We argue that a detector incapable of identifying how a video was forged is likely learning the wrong signal.<n>Unlike binary detection, attribution-guided learning imposes a stronger geometric constraint on the shared embedding space.<n>We propose the Attribution-Guided Multimodal Deepfake Detection framework, which jointly learns to detect and attribute manipulation.
  arXiv  Detail & Related papers  (2026-04-29T09:11:13Z)
• LogitDynamics: Reliable ViT Error Detection from Layerwise Logit Trajectories [1.4610038284393168]
  We study error prediction in Vision Transformers (ViTs)<n>Motivated by internal-signal hallucination detection in large language models, we investigate whether similar depth-wise signals exist inViTs.<n>By attaching lightweight linear heads to intermediate layers, we extract features from the last L layers that capture both the logits of the predicted class and its top-K competitors.<n>A linear probe trained on these features predicts the error indicator.
  arXiv  Detail & Related papers  (2026-04-12T13:43:40Z)
• Traces of Image Memorability in Vision Encoders: Activations, Attention Distributions and Autoencoder Losses [5.369009163979958]
  This paper explores the correlates of image memorability in pretrained vision encoders.<n>We find that these features correlate with memorability to some extent.<n>Results shed light on the relationship between model-internal features and memorability.
  arXiv  Detail & Related papers  (2025-09-01T13:11:59Z)
• Processing and acquisition traces in visual encoders: What does CLIP know about your camera? [28.34664538014526]
  Prior work has analyzed the robustness of visual encoders to image transformations and corruptions.<n>We take a different perspective by analyzing parameters of the image acquisition process and transformations that may be subtle or even imperceptible to the human eye.
  arXiv  Detail & Related papers  (2025-08-14T13:34:13Z)
• Crane: Context-Guided Prompt Learning and Attention Refinement for Zero-Shot Anomaly Detection [50.343419243749054]
  Anomaly detection is critical in fields such as medical diagnostics and industrial defect detection.<n> CLIP's coarse-grained image-text alignment limits localization and detection performance for fine-grained anomalies.<n>Crane improves the state-of-the-art ZSAD from 2% to 28%, at both image and pixel levels, while remaining competitive in inference speed.
  arXiv  Detail & Related papers  (2025-04-15T10:42:25Z)
• Understanding and Improving Training-Free AI-Generated Image Detections with Vision Foundation Models [68.90917438865078]
  Deepfake techniques for facial synthesis and editing pose serious risks for generative models.<n>In this paper, we investigate how detection performance varies across model backbones, types, and datasets.<n>We introduce Contrastive Blur, which enhances performance on facial images, and MINDER, which addresses noise type bias, balancing performance across domains.
  arXiv  Detail & Related papers  (2024-11-28T13:04:45Z)
• Fine-grained Recognition with Learnable Semantic Data Augmentation [68.48892326854494]
  Fine-grained image recognition is a longstanding computer vision challenge. We propose diversifying the training data at the feature-level to alleviate the discriminative region loss problem. Our method significantly improves the generalization performance on several popular classification networks.
  arXiv  Detail & Related papers  (2023-09-01T11:15:50Z)
• UIA-ViT: Unsupervised Inconsistency-Aware Method based on Vision Transformer for Face Forgery Detection [52.91782218300844]
  We propose a novel Unsupervised Inconsistency-Aware method based on Vision Transformer, called UIA-ViT. Due to the self-attention mechanism, the attention map among patch embeddings naturally represents the consistency relation, making the vision Transformer suitable for the consistency representation learning.
  arXiv  Detail & Related papers  (2022-10-23T15:24:47Z)
• Steganalysis of Image with Adaptively Parametric Activation [2.4201849657206496]
  Adap-tive Parametric Activation Module is designed to reserve nega-tive embedding signal. A loss function based on contrastive learning is applied to overcome the limitations of cross-entropy loss by maximum inter-class distance.
  arXiv  Detail & Related papers  (2022-03-24T04:44:51Z)
• Correlation-Aware Deep Tracking [83.51092789908677]
  We propose a novel target-dependent feature network inspired by the self-/cross-attention scheme. Our network deeply embeds cross-image feature correlation in multiple layers of the feature network. Our model can be flexibly pre-trained on abundant unpaired images, leading to notably faster convergence than the existing methods.
  arXiv  Detail & Related papers  (2022-03-03T11:53:54Z)
• Learning Hierarchical Graph Representation for Image Manipulation Detection [50.04902159383709]
  The objective of image manipulation detection is to identify and locate the manipulated regions in the images. Recent approaches mostly adopt the sophisticated Convolutional Neural Networks (CNNs) to capture the tampering artifacts left in the images. We propose a hierarchical Graph Convolutional Network (HGCN-Net), which consists of two parallel branches.
  arXiv  Detail & Related papers  (2022-01-15T01:54:25Z)
• Data Augmentation for Object Detection via Differentiable Neural Rendering [71.00447761415388]
  It is challenging to train a robust object detector when annotated data is scarce. Existing approaches to tackle this problem include semi-supervised learning that interpolates labeled data from unlabeled data. We introduce an offline data augmentation method for object detection, which semantically interpolates the training data with novel views.
  arXiv  Detail & Related papers  (2021-03-04T06:31:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.