Steering Vision-Language Models with Joint Sparse Autoencoders

• URL: http://arxiv.org/abs/2606.25657v1
• Date: Wed, 24 Jun 2026 10:11:04 GMT
• Title: Steering Vision-Language Models with Joint Sparse Autoencoders
• Abstract summary: We introduce the Joint Sparse Autoencoder (JSAE) to factorize sequence-pooled vision and language activations into shared, interpretable image/caption-level features.<n>Applying JSAE to vision-language models (VLMs) yields representations that are difficult to use as controllable cross-modal steering directions.
• Score: 2.367986397868328
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Sparse Autoencoders (SAEs) have shown promise for analyzing language models, but applying them to vision-language models (VLMs) often yields representations that are difficult to use as controllable cross-modal steering directions. We introduce the Joint Sparse Autoencoder (JSAE), which uses an explicit alignment constraint to jointly factorize sequence-pooled vision and language activations into shared, interpretable image/caption-level features. Applied to LLaVA, JSAE recovers cross-modal features for recognizable concepts (e.g., food and animals). Through bidirectional interventions (additive steering and suppression), we observe a layer-dependent asymmetry under our protocol: additive steering peaks at mid-to-late (pre-output) layers and weakens at both ends, whereas suppression scores remain within a comparable range across all probed layers within statistical noise. Experiments on three VLMs, namely LLaVA-v1.6-Mistral-7B, Llama3-LLaVA-8B, and the MoE-based Qwen3-VL-30B, show related layer-localized effects across architectures. Together, these results suggest that explicitly aligned sparse representations support more controllable intervention-based analysis of multimodal features, within an identifiable layer range, than the unconstrained alternatives tested here.

Related papers

• SeGDeP: Semantic- and Geometric-Aware Decoupled Prompts for Reasoning Segmentation [51.46828526392219]
  Reasoning segmentation converts an implicit linguistic conclusion into a precise mask.<n>Existing MLLM-segmenter interfaces either use a special trigger or compress both signals into one context.<n>We present SeGDeP, an explicit what-where interface.
  arXiv  Detail & Related papers  (2026-09-08T15:11:32Z)
• Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Models [12.293070202366316]
  Large Vision-Language Models (LVLMs) achieve strong performance across many multimodal tasks.<n>They often exploit spurious object-background robustness, resulting in predictions driven by contextual shortcuts rather than object-relevant visual evidence.<n>We introduce PURGE, a framework for constructing, benchmarking, and mitigating spurious-correlation-induced failures in LVLMs.
  arXiv  Detail & Related papers  (2026-08-30T19:42:47Z)
• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• Through the LENS: Local Geometric Decomposition of Vision-Language Model Representations [0.5872014229110214]
  LENS (Local Explanation of Neighborhood Subspaces) decomposes VLM activations into local low-rank Gaussian neighborhoods.<n>An automated multimodal labeling pipeline assigns concise semantic descriptions to these neighborhoods.<n>Interpolating activations toward neighborhood centroids causally redirects generation within and across modalities.
  arXiv  Detail & Related papers  (2026-08-01T09:54:54Z)
• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning [99.49739831030388]
  We study how the design of this interface shapes the agent's capacity for open-ended spatial reasoning.<n>We propose SpatialClaw, a training-free framework for spatial reasoning that adopts code as the action interface.
  arXiv  Detail & Related papers  (2026-06-11T17:59:36Z)
• Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models [35.34635490741956]
  Visual latent reasoning lets a multimodal large language model (MLLM) create intermediate visual evidence as continuous tokens.<n>We propose textbfGAP, a textbfGranular textbfAlignment textbfParadigm for visual latent modeling.<n>On Qwen2.5-VL 7B, the resulting model achieves the best mean aggregate perception and reasoning performance among our supervised variants.
  arXiv  Detail & Related papers  (2026-05-12T16:41:09Z)
• MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models [70.34265674686516]
  Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space.<n>We introduce MMEB-V3, a benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios.<n>We conduct a systematic analysis of full-modality embeddings and identify three key findings.
  arXiv  Detail & Related papers  (2026-04-25T14:15:05Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion [91.35078719566472]
  Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection.<n>We introduce Cross-Layer Injection (CLI), a novel and lightweight framework that forges a dynamic many-to-many bridge between the two modalities.
  arXiv  Detail & Related papers  (2026-01-15T18:59:10Z)
• HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Models [4.099133096025821]
  Vision-language models (VLMs) enable open-ended visual question answering but remain prone to hallucinations.<n>We present HEDGE, a unified framework for hallucination detection that combines controlled visual perturbations, semantic clustering, and robust uncertainty metrics.
  arXiv  Detail & Related papers  (2025-11-16T17:16:31Z)
• IUT-Plug: A Plug-in tool for Interleaved Image-Text Generation [23.61167100602915]
  IUT-Plug is a module grounded in an Image Understanding Tree (IUT)<n>A dynamic IUT-Plug extraction module parses visual scenes into hierarchical symbolic structures.<n>A coordinated narrative-flow and image synthesis mechanism ensures cross-modal consistency.
  arXiv  Detail & Related papers  (2025-10-13T03:19:45Z)
• Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation [59.40886078302025]
  Multimodal large language models (MLLMs) have demonstrated remarkable capabilities in aligning visual inputs with natural language outputs.<n>Yet, the extent to which generated tokens depend on visual modalities remains poorly understood.<n>We present a lightweight black-box framework for explaining autoregressive token generation in MLLMs.
  arXiv  Detail & Related papers  (2025-09-26T15:38:42Z)
• GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs [56.93583799109029]
  GrAInS is an inference-time steering approach that operates across both language-only and vision-language models and tasks.<n>During inference, GrAInS hidden activations at transformer layers guided by token-level attribution signals, and normalizes activations to preserve representational scale.<n>It consistently outperforms both fine-tuning and existing steering baselines.
  arXiv  Detail & Related papers  (2025-07-24T02:34:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.