Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression

• URL: http://arxiv.org/abs/2608.02109v1
• Date: Mon, 03 Aug 2026 12:08:56 GMT
• Title: Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression
• Abstract summary: Vision-Text Compression (VTC) renders long texts into images and encodes them through the vision encoder (ViT)<n>ViT captures visual attributes (glyphs, font sizes, layout) rather than linguistic semantics, causing rendered-image representations to diverge from native-text representations.<n>We propose SPIRAL, a self-supervised alignment framework that closes this gap using only the model's own text-path behavior as supervision.
• Score: 7.944386069136441
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Text Compression (VTC) renders long texts into images and encodes them through the vision encoder (ViT), compressing thousands of text tokens into far fewer visual tokens. However, since the ViT is pretrained predominantly on natural images, it captures visual attributes (glyphs, font sizes, layout) rather than linguistic semantics, causing rendered-image representations to diverge from native-text representations. We term this cross-path inconsistency and show, via rendering perturbation experiments, that it is a critical yet overlooked bottleneck of VTC. We propose SPIRAL (Self-improving Path Integration and Realignment), a self-supervised alignment framework that closes this gap using only the model's own text-path behavior as supervision, requiring no external teachers or additional annotations. SPIRAL operates at two complementary granularities: token-level on-policy distillation (OPD) for local faithfulness, and sequence-level preference optimization (DPO) for global coherence. On VTCBench, SPIRAL improves the overall score of Qwen3-VL-8B from 35.10 to 54.02, approaching the native text-input performance (55.60) and outperforming models up to 30x larger. The two granularities exhibit complementary strengths: OPD excels at retrieval and is sample-efficient, while DPO is stronger on reasoning and memory and scales better with data. SPIRAL's benefits also generalize to out-of-domain benchmarks, confirming that effective VTC hinges on aligning rendered-image representations back to native-text semantics.

Related papers

• Full Glyph Images Beat Token Embeddings: A Controlled Study for Transformers [1.920312875157821]
  We use a Vision-based model and an index-based baseline to represent Chinese characters.<n>Across all tested decoder backbones, the Vision-based model consistently outperforms the baseline.<n>The advantage requires both spatially coherent input and a ViT encoder with 2D positional encodings.
  arXiv  Detail & Related papers  (2026-07-04T19:30:16Z)
• TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment [87.69757277913577]
  We propose TEVI, a framework that uses captions as a signal for what to retain from image embeddings.<n>In a controlled setup with synthetic captions, we show that TEVI is effective at preserving caption-described attributes while discarding others.
  arXiv  Detail & Related papers  (2026-06-05T16:54:40Z)
• Pixel-TTS: Image based Text Rendering for Robust Text-to-Speech [3.62994537177299]
  Grounding text in its visual form allows structurally similar characters with different Unicode encodings to produce similar embeddings.<n>We propose Pixel-TTS, the first framework for visually grounded speech synthesis.<n>It renders text as images and projects them through a 2D convolutional layer to generate embeddings.
  arXiv  Detail & Related papers  (2026-06-05T11:46:30Z)
• PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation [70.26672362704758]
  We introduce PixelRAG, a new retrieval-augmented method that represents websites in their native visual form.<n> PixelRAG is the first pipeline to operate over a full Wikipedia corpus in this form.<n>It consistently outperforms both no-retrieval and text-based RAG baselines.
  arXiv  Detail & Related papers  (2026-06-01T23:20:51Z)
• DV-SFT: Direct Vision Supervision for Fine-Grained Visual Understanding [14.240098378416995]
  Multimodal large language models are typically trained end-to-end to predict ground-truth answers, yet supervision signals are applied exclusively to text tokens.<n>Visual tokens, the core carriers of visual information, are optimized only implicitly as part of the context, leading to coarse-grained visual understanding.<n>We propose textbfDirect textbfVision textbfSupervised textbfFine-textbfTuning (DV-SFT), which constructs explicit, token-level supervision for visual tokens and trains
  arXiv  Detail & Related papers  (2026-05-26T07:41:48Z)
• LensVLM: Selective Context Expansion for Compressed Visual Representation of Text [43.63292785228071]
  Vision Language Models (VLMs) can process text as rendered images, bypassing the need for tokenizing the text.<n>We propose LensVLM, an inference framework and post-training recipe that enables VLMs to scan compressed images, then selectively expand only the relevant images.<n>Building on Qwen3.5-9B-Base, LensVLM maintains accuracy comparable to the full-text upper bound at 4.3x effective compression.<n>LensVLM also generalizes to multimodal document and code understanding tasks, with the accuracy gain over baselines growing as compression increases.
  arXiv  Detail & Related papers  (2026-05-07T23:03:21Z)
• VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text? [51.02924254085878]
  Vision-Language Models (VLMs) have achieved impressive performance in cross-modal understanding across textual and visual inputs.<n>We introduce VISTA-Bench, a benchmark from multimodal perception, reasoning, to unimodal understanding domains.
  arXiv  Detail & Related papers  (2026-02-04T17:48:55Z)
• $β$-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language Alignment [53.42377319350806]
  $$-CLIP is a multi-granular text-conditioned contrastive learning framework.<n>$$-CAL addresses the semantic overlap inherent in this hierarchy.<n>$$-CLIP establishes a robust, adaptive baseline for dense vision-language correspondence.
  arXiv  Detail & Related papers  (2025-12-14T13:03:20Z)
• Understanding the Effect of using Semantically Meaningful Tokens for Visual Representation Learning [40.08368469646114]
  We provide semantically-meaningful visual tokens to transformer encoders within a vision-language pre-training framework.<n>We demonstrate notable improvements over ViTs in learned representation quality across text-to-image and image-to-text retrieval tasks.
  arXiv  Detail & Related papers  (2024-05-26T01:46:22Z)
• Story Visualization by Online Text Augmentation with Context Memory [64.86944645907771]
  We propose a novel memory architecture for the Bi-directional Transformer framework with an online text augmentation. The proposed method significantly outperforms the state of the arts in various metrics including FID, character F1, frame accuracy, BLEU-2/3, and R-precision.
  arXiv  Detail & Related papers  (2023-08-15T05:08:12Z)
• TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection [61.0662744915659]
  We propose an efficient vision-and-language pre-training model with textbfText-textbfRelevant textbfImage textbfPatch textbfSelection, namely TRIPS.<n> TRIPS reduces the visual sequence progressively with a text-guided patch-selection layer in the visual backbone for efficient training and inference.
  arXiv  Detail & Related papers  (2023-05-08T05:53:30Z)
• Visually-augmented pretrained language models for NLP tasks without images [77.74849855049523]
  Existing solutions often rely on explicit images for visual knowledge augmentation. We propose a novel textbfVisually-textbfAugmented fine-tuning approach. Our approach can consistently improve the performance of BERT, RoBERTa, BART, and T5 at different scales.
  arXiv  Detail & Related papers  (2022-12-15T16:13:25Z)
• Self-supervised Character-to-Character Distillation for Text Recognition [54.12490492265583]
  We propose a novel self-supervised Character-to-Character Distillation method, CCD, which enables versatile augmentations to facilitate text representation learning. CCD achieves state-of-the-art results, with average performance gains of 1.38% in text recognition, 1.7% in text segmentation, 0.24 dB (PSNR) and 0.0321 (SSIM) in text super-resolution.
  arXiv  Detail & Related papers  (2022-11-01T05:48:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.