TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment

• URL: http://arxiv.org/abs/2606.07451v1
• Date: Fri, 05 Jun 2026 16:54:40 GMT
• Title: TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment
• Abstract summary: We propose TEVI, a framework that uses captions as a signal for what to retain from image embeddings.<n>In a controlled setup with synthetic captions, we show that TEVI is effective at preserving caption-described attributes while discarding others.
• Score: 87.69757277913577
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models such as CLIP are highly useful for diverse tasks due to their shared image-text embedding space. Despite this, the image and text embeddings are often poorly aligned, affecting downstream performance. Recent work has shown that this can be attributed to an information imbalance: images contain more information than their captions describe. In this work, we propose TEVI, a framework that uses captions as a signal for what to retain from image embeddings. Specifically, we use sparse autoencoders to disentangle image embeddings and train a masking module to selectively reconstruct the embedding based on a given caption. In a controlled setup with synthetic captions, we show that TEVI is effective at preserving caption-described attributes while discarding others. By applying TEVI to CLIP models trained on natural images, we further achieve improved retrieval performance across coarse-grained short-caption (MS COCO, Flickr) and fine-grained long-caption (IIW, DOCCI) benchmarks, with stronger gains on richer captions, and improved robustness on the RoCOCO benchmark.

Related papers

• CycleCap: Improving VLMs Captioning Performance via Self-Supervised Cycle Consistency Fine-Tuning [65.10059440725041]
  Visual-Language Models (VLMs) have achieved remarkable progress in image captioning, visual question answering, and visual reasoning.<n>They remain prone to vision-language misalignment, often producing overly generic or hallucinated descriptions.<n>Existing approaches address this via instruction tuning-requiring costly, large-scale annotated datasets or via complex test-time frameworks for caption refinement.<n>In this work, we revisit image-text alignment through the lens of cycle consistency.
  arXiv  Detail & Related papers  (2026-03-18T20:57:31Z)
• CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions [31.624782806591682]
  We introduce two simple yet effective designs to better leverage richly described synthetic captions. First, we observe a strong inverse effect in learning with synthetic captions. Second, we incorporate an autoregressive captioner to mimic the recaptioning process.
  arXiv  Detail & Related papers  (2024-11-25T18:49:02Z)
• Decoder Pre-Training with only Text for Scene Text Recognition [54.93037783663204]
  Scene text recognition (STR) pre-training methods have achieved remarkable progress, primarily relying on synthetic datasets. We introduce a novel method named Decoder Pre-training with only text for STR (DPTR) DPTR treats text embeddings produced by the CLIP text encoder as pseudo visual embeddings and uses them to pre-train the decoder.
  arXiv  Detail & Related papers  (2024-08-11T06:36:42Z)
• Learning text-to-video retrieval from image captioning [59.81537951811595]
  We describe a protocol to study text-to-video retrieval training with unlabeled videos. We assume (i) no access to labels for any videos, and (ii) access to labeled images in the form of text. We show that automatically labeling video frames with image captioning allows text-to-video retrieval training.
  arXiv  Detail & Related papers  (2024-04-26T15:56:08Z)
• VIXEN: Visual Text Comparison Network for Image Difference Captioning [58.16313862434814]
  We present VIXEN, a technique that succinctly summarizes in text the visual differences between a pair of images. Our proposed network linearly maps image features in a pairwise manner, constructing a soft prompt for a pretrained large language model.
  arXiv  Detail & Related papers  (2024-02-29T12:56:18Z)
• Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training [14.340740609933437]
  We propose a novel zero-shot image captioning framework with text-only training to reduce the modality gap. In particular, we introduce a subregion feature aggregation to leverage local region information. We extend our framework to build a zero-shot VQA pipeline, demonstrating its generality.
  arXiv  Detail & Related papers  (2024-01-04T16:43:46Z)
• FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions [11.274127953112574]
  We propose an automated approach to augmenting existing captions with visual details using "frozen" vision experts. Our proposed method, FuseCap, fuses the outputs of such vision experts with the original captions using a large language model. We release this large-scale dataset of enriched image-caption pairs for the community.
  arXiv  Detail & Related papers  (2023-05-28T13:16:03Z)
• Retrieval-augmented Image Captioning [15.266569206458648]
  We present a new approach to image captioning that generates sentences given the input image and a set of captions retrieved from a datastore. The encoder in our model jointly processes the image and retrieved captions using a pretrained V&L BERT. Our work contributes towards using pretrained V&L encoders for generative tasks, instead of standard classification tasks.
  arXiv  Detail & Related papers  (2023-02-16T12:54:13Z)
• Fine-grained Image Captioning with CLIP Reward [104.71533106301598]
  We propose using CLIP, a multimodal encoder trained on huge image-text pairs from web, to calculate multimodal similarity and use it as a reward function. We also propose a simple finetuning strategy of the CLIP text encoder to improve grammar that does not require extra text annotation. In experiments on text-to-image retrieval and FineCapEval, the proposed CLIP-guided model generates more distinctive captions than the CIDEr-optimized model.
  arXiv  Detail & Related papers  (2022-05-26T02:46:09Z)
• CapOnImage: Context-driven Dense-Captioning on Image [13.604173177437536]
  We introduce a new task called captioning on image (CapOnImage), which aims to generate dense captions at different locations of the image based on contextual information. We propose a multi-modal pre-training model with multi-level pre-training tasks that progressively learn the correspondence between texts and image locations. Compared with other image captioning model variants, our model achieves the best results in both captioning accuracy and diversity aspects.
  arXiv  Detail & Related papers  (2022-04-27T14:40:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.