Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders

• URL: http://arxiv.org/abs/2606.29888v1
• Date: Mon, 29 Jun 2026 07:27:24 GMT
• Title: Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders
• Abstract summary: Vision-language models map images and text into a joint embedding space.<n>These embeddings often entangle multiple semantic features, which limits their interpretability and controllability.<n>We propose an approach that trains modality-specific sparse autoencoders to preserve each modality's feature geometry, and then aligns corresponding features post hoc.
• Score: 11.191047696392026
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Vision-language models map images and text into a joint embedding space. However, these embeddings often entangle multiple semantic features, which limits their interpretability and controllability. While sparse autoencoders have emerged as a useful tool for decomposing these embeddings into monosemantic features, their application to joint embedding spaces has largely relied on an implicit, untested assumption that semantically corresponding features share the same directions across modalities. In this paper, we challenge this assumption by identifying discrepancies in feature directions for the same concept across image and text modalities, a phenomenon we term cross-modal feature heterogeneity. We demonstrate that this heterogeneity is a key driver of the modality split, where a shared concept activates different latents depending on the modality. This finding further reveals why aligning latent activations alone is insufficient to resolve the underlying feature mismatch. Motivated by this observation, we propose an approach that trains modality-specific sparse autoencoders to preserve each modality's feature geometry, and then aligns corresponding features post hoc. Our method improves reconstruction fidelity and enhances performance in cross-modal retrieval and concept steering.

Related papers

• Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering [7.62772056485722]
  We introduce emphcross-branch semantic steering, an intervention-based framework that extracts semantic directions from one branch and applies them to the other.<n>We show that steering vectors learned from the understanding branch can transfer to generation, enabling controllable image synthesis and improved semantic faithfulness.<n>Our results reveal that architectural unification does not guarantee semantic alignment, and establish cross-branch steering as a practical tool for probing multimodal representations.
  arXiv  Detail & Related papers  (2026-07-29T02:46:20Z)
• When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities [10.807285909852277]
  We propose a Structured Sparse AutoEncoder ($S2AE$) that enforces concept consistency from both semantic and spatial perspectives in the visual modality.<n>We group image patches based on Transformer attention similarity and spatial proximity, and introduce a structured sparsity regularization when training the vanilla SAE.<n>The regularization consists of exclusive sparsity for inter-group concept disentanglement and group sparsity for intra-group concept consistency, which drives the latent neurons by SAEs to specialize in distinct, semantically grounded concepts.
  arXiv  Detail & Related papers  (2026-07-09T15:35:08Z)
• Variational Adapter for Cross-modal Similarity Representation [2.0621012870194755]
  We propose a Variational Adapter for Cross-modal Similarity Representation (VACSR)<n>It reformulates image-text matching with fine-grained semantic scarcity as a variational inference problem.<n>It constructs a latent space for cross-modal similarity and uses regularization techniques to mitigate overfitting to binary annotations.
  arXiv  Detail & Related papers  (2026-05-29T08:08:15Z)
• Semantic Optimal Transport for Sparse Autoencoder Feature Matching and Circuit Compression [11.444432919120173]
  Two key SAE analyses that remain difficult to scale are matching semantically similar features across multi-layers and compressing large feature circuits into interpretable supernodes.<n>We introduce a distributional framework for this problem, in which each feature is represented not by a single decoder vector like in the literature, but by an activation-weighted distribution over the hidden states that express it.<n>We prove that our representation is invariant to activation rescaling, stable under perturbations, and recovers true matches under finite-sample margin conditions.
  arXiv  Detail & Related papers  (2026-05-27T14:54:58Z)
• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Anisotropic Modality Align [91.23979617826926]
  Training multimodal large language models has long been limited by the scarcity of high-quality paired multimodal data.<n>Recent studies show that the shared representation space of pretrained multimodal contrastive models can serve as a bridge, enabling models to perform multimodal training with unimodal data.<n>The core obstacle lies in the persistent Modality Gap in the shared space.
  arXiv  Detail & Related papers  (2026-05-08T14:53:24Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• Interpreting the linear structure of vision-language model embedding spaces [12.846590038965774]
  We train and release sparse autoencoders (SAEs) on the embedding spaces of four vision-language models.<n>SAEs approximate model embeddings as sparse linear combinations of learned directions, or "concepts"<n>Retraining SAEs with different seeds or different data diet leads to two findings: the rare, specific concepts captured by the SAEs are liable to change drastically, but we also show that commonly-activating concepts are remarkably stable across runs.
  arXiv  Detail & Related papers  (2025-04-16T01:40:06Z)
• GSSF: Generalized Structural Sparse Function for Deep Cross-modal Metric Learning [51.677086019209554]
  We propose a Generalized Structural Sparse to capture powerful relationships across modalities for pair-wise similarity learning. The distance metric delicately encapsulates two formats of diagonal and block-diagonal terms. Experiments on cross-modal and two extra uni-modal retrieval tasks have validated its superiority and flexibility.
  arXiv  Detail & Related papers  (2024-10-20T03:45:50Z)
• Image Synthesis via Semantic Composition [74.68191130898805]
  We present a novel approach to synthesize realistic images based on their semantic layouts. It hypothesizes that for objects with similar appearance, they share similar representation. Our method establishes dependencies between regions according to their appearance correlation, yielding both spatially variant and associated representations.
  arXiv  Detail & Related papers  (2021-09-15T02:26:07Z)
• Exploring Modality-shared Appearance Features and Modality-invariant Relation Features for Cross-modality Person Re-Identification [72.95858515157603]
  Cross-modality person re-identification works rely on discriminative modality-shared features. Despite some initial success, such modality-shared appearance features cannot capture enough modality-invariant information. A novel cross-modality quadruplet loss is proposed to further reduce the cross-modality variations.
  arXiv  Detail & Related papers  (2021-04-23T11:14:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.