Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders

• URL: http://arxiv.org/abs/2607.06445v1
• Date: Tue, 07 Jul 2026 16:11:13 GMT
• Title: Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders
• Abstract summary: Vision-Language Models (VLMs) are increasingly utilized as the conditioning backbone for diffusion-based image editing.<n>While standalone VLMs demonstrate strong localization capabilities, editing pipelines frequently struggle to maintain this accuracy.<n>In this work, we investigate this performance gap, hypothesizing that it stems from treating the VLM as a condition encoder.
• Score: 54.06522068041745
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) are increasingly utilized as the conditioning backbone for diffusion-based image editing due to their remarkable multimodal reasoning capabilities. While standalone VLMs demonstrate strong localization capabilities, editing pipelines frequently struggle to maintain this accuracy, particularly in complex, multi-entity scenes. In this work, we investigate this performance gap, hypothesizing that it stems from treating the VLM as a condition encoder. In this role, the model is restricted to a single forward pass, preventing the autoregressive generation process for which it was optimized, thereby failing to fully expose its capabilities. To investigate whether this spatial understanding persists when the VLM is used as a condition encoder, we introduce Analysis-by-Proxy. In this framework, we train a lightweight, interpretable proxy model on the VLM's intermediate representations using an auxiliary localization task. By analyzing the VLM through this proxy, we uncover the specific VLM representations that encode localization information. Our findings expose a fundamental mismatch between how spatial knowledge is represented within a VLM condition encoder and how it is extracted by current editing pipelines. We reveal that under single-pass constraints, the localization signal does not reliably propagate to the predefined layer configurations commonly used for conditioning. Instead, this crucial signal remains hidden within intermediate representations, at locations that vary depending on the input prompt. Using our introduced Analysis-by-Proxy framework, we reveal the fundamental failures of existing condition extraction strategies in editing pipelines, opening the door to more principled design of conditioning architectures.

Related papers

• Continual Video-MLLM Adaptation over Evolving Domains [42.6837206153246]
  Video multimodal large language models have shown strong capability in video understanding, yet their adaptation to sequentially evolving domains remains underexplored.<n>We propose Distribution-Aware Expert Routing, a parameter-efficient framework for continual Video-MLLM adaptation over evolving domains.<n> Experiments on two strong Video-MLLM backbones show that DAER consistently outperforms prior methods.
  arXiv  Detail & Related papers  (2026-07-21T05:16:36Z)
• What Semantics Survive the Connector? Diagnosing VLM-to-DiT Alignment in Video Editing [59.40855219373921]
  Flow matching based video generative models rely on prepended Vision-Language Models to handle complex, instruction-based video editing.<n>The prevailing assumption is that a connector module can seamlessly align the VLM's rich multi-modal reasoning with the original text embedding space of DiTs.<n>We propose a controlled data processing pipeline based on video composition that results in TRACE-Edit, a diagnostic dataset focusing on relation-based editing.
  arXiv  Detail & Related papers  (2026-05-20T06:42:15Z)
• Domain Restriction via Multi SAE Layer Transitions [0.20052993723676893]
  We show that layer transitions provide a promising avenue for extracting domain-specific signature.<n>Specifically, we present several lightweight ways of learning on internal dynamics encoded using a sparse autoencoder.<n>We provide a comprehensive analysis of the method and benchmark it with the gemma-2 2B and 9B models.
  arXiv  Detail & Related papers  (2026-05-12T10:36:02Z)
• Where to Place the Query? Unveiling and Mitigating Positional Bias in In-Context Learning for Diffusion LLMs via Decoding Dynamics [5.345984783628041]
  This paper presents a comprehensive analysis unveiling that query position is actually a first-order variable in dLLMs.<n>We propose Average Confidence ($overlineC$), a novel metric tracking the iterative decoding process.<n>By establishing the foundational spatial ICL baselines, we introduce Auto-ICL, a training-free adaptive routing strategy.
  arXiv  Detail & Related papers  (2026-04-26T20:22:47Z)
• Rethinking VLMs for Image Forgery Detection and Localization [55.32700985102152]
  We study how to fully leverage vision-language models (VLMs) to assist the image forgery detection and localization (IFDL) task.<n>Building on these findings, we propose a new IFDL pipeline named IFDL-VLM.<n>The experimental results show that we consistently achieve new state-of-the-art performance in detection, localization, and interpretability.
  arXiv  Detail & Related papers  (2026-03-13T12:21:31Z)
• Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detection [52.5174167737992]
  Video anomaly detection (VAD) aims to identify abnormal events in videos.<n>We propose SteerVAD, which advances MLLM-based VAD by shifting from passively reading to actively steering and rectifying internal representations.<n>Our method achieves state-of-the-art performance among tuning-free approaches requiring only 1% of training data.
  arXiv  Detail & Related papers  (2026-02-27T13:48:50Z)
• EPRBench: A High-Quality Benchmark Dataset for Event Stream Based Visual Place Recognition [54.55914886780534]
  Event stream-based Visual Place Recognition (VPR) is an emerging research direction that offers a compelling solution to the instability of conventional visible-light cameras under challenging conditions such as low illumination, overexposure, and high-speed motion.<n>We introduce EPRBench, a high-quality benchmark specifically designed for event stream-based VPR.<n>EPRBench comprises 10K event sequences and 65K event frames, collected using both handheld and vehicle-mounted setups to comprehensively capture real-world challenges across diverse viewpoints, weather conditions, and lighting scenarios.
  arXiv  Detail & Related papers  (2026-02-13T13:25:05Z)
• Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models [16.73630874846666]
  Visual/geometrical and textual representations of spatial structure must be combined at some point in Vision Language Models (VLMs)<n>We show thatVLMs encode object locations by linearly binding itspatialtemporal IDs to textual activations, then perform reasoning via language tokens.<n>We extend our analysis to video VLMs and identify an analogous linear temporal ID mechanism.
  arXiv  Detail & Related papers  (2026-01-18T23:48:38Z)
• On the Reliability of Vision-Language Models Under Adversarial Frequency-Domain Perturbations [53.611451075703314]
  Vision-Language Models (VLMs) are increasingly used as perceptual modules for visual content reasoning.<n>We show how these feature transformations undermine authenticity/DeepFake detection and automated image captioning tasks.
  arXiv  Detail & Related papers  (2025-07-30T05:41:29Z)
• Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor [32.34399128209528]
  We study whether pre-trained text-to-image diffusion models can serve as instruction-aware visual encoders.<n>We find diffusion features are both rich in semantics and can encode strong image-text alignment.<n>We then investigate how to align these features with large language models and uncover a leakage phenomenon.
  arXiv  Detail & Related papers  (2025-07-09T17:59:47Z)
• Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval [23.94611751368491]
  We investigate the feasibility of leveraging large language models (LLMs) for integrating general knowledge and incorporating pseudo-events as priors for temporal content distribution. To overcome these limitations, we propose utilizing LLM encoders instead of decoders. We present a general framework for integrating LLM encoders into existing VMR architectures, specifically within the fusion module.
  arXiv  Detail & Related papers  (2024-07-21T04:39:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.