LensVLM: Selective Context Expansion for Compressed Visual Representation of Text

• URL: http://arxiv.org/abs/2605.07019v1
• Date: Thu, 07 May 2026 23:03:21 GMT
• Title: LensVLM: Selective Context Expansion for Compressed Visual Representation of Text
• Abstract summary: Vision Language Models (VLMs) can process text as rendered images, bypassing the need for tokenizing the text.<n>We propose LensVLM, an inference framework and post-training recipe that enables VLMs to scan compressed images, then selectively expand only the relevant images.<n>Building on Qwen3.5-9B-Base, LensVLM maintains accuracy comparable to the full-text upper bound at 4.3x effective compression.<n>LensVLM also generalizes to multimodal document and code understanding tasks, with the accuracy gain over baselines growing as compression increases.
• Score: 43.63292785228071
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision Language Models (VLMs) offer the exciting possibility of processing text as rendered images, bypassing the need for tokenizing the text into long token sequences. Since VLM image encoders map fixed-size images to a fixed number of visual tokens, varying rendering resolution provides a fine-grained compression knob. However, accuracy deteriorates quickly as compression increases: characters shrink below the vision encoder's effective resolution, making them indistinguishable. To address this, we propose LensVLM, an inference framework and post-training recipe that enables VLMs to scan compressed images, then selectively expand only the relevant images to their uncompressed form via learned tools. Building on Qwen3.5-9B-Base, LensVLM maintains accuracy comparable to the full-text upper bound at 4.3x effective compression and outperforms retrieval-based, text- and visual-compression baselines up to 10.1x effective compression across seven text QA benchmarks. LensVLM also generalizes to multimodal document and code understanding tasks, with the accuracy gain over baselines growing as compression increases. Our analysis validates this approach: training makes visual compression robust to rendering choices, and as compression grows the model increasingly relies on expanded content rather than unreliable visual reading. The analysis also yields practical tool-choice guidance: text expansion is preferable for rendered text, while high-resolution image expansion suits native documents whose layout cues carry task-relevant information.

Related papers

• Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression [7.944386069136441]
  Vision-Text Compression (VTC) renders long texts into images and encodes them through the vision encoder (ViT)<n>ViT captures visual attributes (glyphs, font sizes, layout) rather than linguistic semantics, causing rendered-image representations to diverge from native-text representations.<n>We propose SPIRAL, a self-supervised alignment framework that closes this gap using only the model's own text-path behavior as supervision.
  arXiv  Detail & Related papers  (2026-08-03T12:08:56Z)
• TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment [87.69757277913577]
  We propose TEVI, a framework that uses captions as a signal for what to retain from image embeddings.<n>In a controlled setup with synthetic captions, we show that TEVI is effective at preserving caption-described attributes while discarding others.
  arXiv  Detail & Related papers  (2026-06-05T16:54:40Z)
• Glyph: Scaling Context Windows via Visual-Text Compression [91.20717058018745]
  Glyph is a framework that renders long texts into images and processes them with vision-language models.<n>Our method achieves 3-4x token compression while maintaining accuracy comparable to leading long-context models.<n>Under extreme compression, a 128K-context VLM could scale to handle 1M-token-level text tasks.
  arXiv  Detail & Related papers  (2025-10-20T17:58:56Z)
• End-to-End Semantic Preservation in Text-Aware Image Compression Systems [42.76781276416154]
  We present an end-to-end compression framework that retains text-specific features for Optical Character Recognition (OCR)<n> Experiments show significant improvements in text extraction accuracy at lows, even outperforming OCR on uncompressed images.<n>We extend this study to general-purpose encoders, exploring their capacity to preserve hidden semantics under extreme compression.
  arXiv  Detail & Related papers  (2025-03-25T09:36:13Z)
• Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformers [58.80845404416028]
  Data-free quantization (DFQ) enables model quantization without accessing real data, addressing concerns regarding data security and privacy.<n>With the growing adoption of Vision Transformers (ViTs), DFQ for ViTs has garnered significant attention.<n>We propose SARDFQ, a novel Semantics Alignment and Reinforcement Data-Free Quantization method for ViTs.
  arXiv  Detail & Related papers  (2024-12-21T09:30:45Z)
• AdaptVision: Dynamic Input Scaling in MLLMs for Versatile Scene Understanding [96.01726275876548]
  We present AdaptVision, a multimodal large language model specifically designed to dynamically process input images at varying resolutions. We devise a dynamic image partitioning module that adjusts the number of visual tokens according to the size and aspect ratio of images. Our model is capable of processing images with resolutions up to $1008times 1008$.
  arXiv  Detail & Related papers  (2024-08-30T03:16:49Z)
• Decoder Pre-Training with only Text for Scene Text Recognition [54.93037783663204]
  Scene text recognition (STR) pre-training methods have achieved remarkable progress, primarily relying on synthetic datasets. We introduce a novel method named Decoder Pre-training with only text for STR (DPTR) DPTR treats text embeddings produced by the CLIP text encoder as pseudo visual embeddings and uses them to pre-train the decoder.
  arXiv  Detail & Related papers  (2024-08-11T06:36:42Z)
• Perceptual Image Compression with Cooperative Cross-Modal Side Information [53.356714177243745]
  We propose a novel deep image compression method with text-guided side information to achieve a better rate-perception-distortion tradeoff. Specifically, we employ the CLIP text encoder and an effective Semantic-Spatial Aware block to fuse the text and image features.
  arXiv  Detail & Related papers  (2023-11-23T08:31:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.