FastOCR: Dynamic Visual Fixation via KV Cache Pruning for Efficient Document Parsing

• URL: http://arxiv.org/abs/2605.17447v1
• Date: Sun, 17 May 2026 13:39:47 GMT
• Title: FastOCR: Dynamic Visual Fixation via KV Cache Pruning for Efficient Document Parsing
• Abstract summary: We propose FastOCR, a training-free framework with two complementary modules.<n>FastOCR retains 98% of the unpruned model's accuracy while attending to only 5% of the visual tokens per decoding step.
• Score: 51.905216364362325
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) have shown strong promise on Optical Character Recognition (OCR), yet the sheer number of visual tokens required to encode dense documents incurs prohibitive inference cost. Existing pruning methods rely on physical eviction, e.g., permanently discarding visual tokens during the prefill stage. While effective for natural images, this strategy fundamentally breaks down on OCR, where virtually every visual token may correspond to a character or structural element, and any irreversible loss leads to catastrophic accuracy degradation. We observe that, although document images appear globally dense and seemingly unprunable, the model's attention to them is in fact temporally sparse: at each decoding step it concentrates on a small region that shifts gradually across steps, much as a human reader fixates on successive words rather than perceiving an entire page at once. Motivated by this Dynamic Visual Fixation phenomenon, we recast the intractable global pruning problem as a tractable local, dynamic one and propose FastOCR, a training-free framework with two complementary modules. Specifically, Focal-Guided Pruning identifies a small set of focal layers and selects the most task-relevant visual tokens from them at each step, while Cross-Step Fixation Reuse exploits the gradual shift of fixation to warm-start each step from the previous one. By dynamically adjusting which tokens are attended rather than evicting any from the cache, FastOCR avoids permanent information loss. Extensive experiments show that FastOCR serves as a plug-and-play acceleration module, generalizing consistently across five VLMs of varying sizes and architectures. On Qwen2.5-VL, FastOCR retains 98% of the unpruned model's accuracy while attending to only 5% of the visual tokens per decoding step, reducing attention latency by 3.0$\times$.

Related papers

• PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference [15.233755808671871]
  Existing visual token pruning methods exhibit two fundamental limitations.<n>Most approaches operate exclusively post-vision encoder, leaving the substantial latency of the visual encoding phase unoptimized.<n>We propose PACE, a framework that accelerates both the vision encoder and the Large Language Model.
  arXiv  Detail & Related papers  (2026-08-27T14:52:09Z)
• LayoutLite: Token-Level Implicit Layout Analysis for Efficient Document OCR [10.208363125551555]
  We propose LayoutLite, a lightweight plug-and-play module for efficient document OCR.<n>It aggregates multi-layer visual representations from the vision encoder, and predicts an importance score for each visual token with a lightweight scoring network.<n>Experiments on OmniDocBench demonstrate that LayoutLite can substantially reduce visual token length and inference cost with negligible degradation in recognition quality.
  arXiv  Detail & Related papers  (2026-07-24T11:13:44Z)
• CaTok: Taming Mean Flows for One-Dimensional Causal Image Tokenization [122.88484422855934]
  We present CaTok, a 1D causal image tokenizer with a MeanFlow decoder.<n>By selecting tokens over time intervals, CaTok learns causal 1D representations that support both fast one-step generation and high-fidelity multi-step sampling.<n> Experiments demonstrate that CaTok achieves state-of-the-art results on ImageNet reconstruction, reaching 0.75 FID, 22.53 PSNR and 0.674 SSIM.
  arXiv  Detail & Related papers  (2026-03-06T16:39:17Z)
• SimpleOCR: Rendering Visualized Questions to Teach MLLMs to Read [43.28273039987167]
  We introduce the Visualized-Question (VQ) setting, where text queries are rendered directly onto images.<n>Despite possessing strong OCR capabilities, models suffer a performance degradation of up to 12.7% in the VQ setting.<n>We propose SimpleOCR, a plug-and-play training strategy that imposes a structural constraint on the learning process.
  arXiv  Detail & Related papers  (2026-02-25T21:36:30Z)
• DODO: Discrete OCR Diffusion Models [15.352694377412229]
  We introduce DODO, the first VLM to utilize block discrete diffusion and unlock its speedup potential for OCR.<n>Our method achieves near state-of-the-art accuracy while enabling up to 3x faster inference compared to autoregressive baselines.
  arXiv  Detail & Related papers  (2026-02-18T20:59:22Z)
• ResTok: Learning Hierarchical Residuals in 1D Visual Tokenizers for Autoregressive Image Generation [64.84095852784714]
  Residual Tokenizer (ResTok) is a 1D visual tokenizer that builds hierarchical residuals for both image tokens and latent tokens.<n>We show that restoring hierarchical residual priors in visual tokenization significantly improves AR image generation, achieving a gFID of 2.34 on ImageNet-256 with only 9 sampling steps.
  arXiv  Detail & Related papers  (2026-01-07T14:09:18Z)
• Fast SAM2 with Text-Driven Token Pruning [52.8350457627401]
  Segment Anything Model 2 (SAM2), a vision computation model has significantly advanced in prompt-driven video object segmentation.<n>SAM2 pipelines propagate all visual tokens produced by the image encoder through downstream temporal reasoning modules, regardless of their relevance to the target object.<n>We introduce a text-guided token pruning framework that improves inference efficiency by selectively reducing token density prior to temporal propagation.
  arXiv  Detail & Related papers  (2025-12-24T18:59:05Z)
• VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models [57.2662376527586]
  VScan is a two-stage visual token reduction framework.<n>It addresses token redundancy by: (1) integrating complementary global and local scans with token merging during visual encoding, and (2) introducing pruning at intermediate layers of the language model.<n>VScan achieves a 2.91$times$ speedup in prefilling and a 10$times$ reduction in FLOPs, while retaining 95.4% of the original performance.
  arXiv  Detail & Related papers  (2025-05-28T17:59:08Z)
• ToDRE: Visual Token Pruning via Diversity and Task Awareness for Efficient Large Vision-Language Models [59.47738955960352]
  ToDRE is a two-stage and training-free token compression framework.<n>It achieves superior performance by pruning tokens based on token Diversity and token-task RElevance.
  arXiv  Detail & Related papers  (2025-05-24T15:47:49Z)
• MARché: Fast Masked Autoregressive Image Generation with Cache-Aware Attention [10.077033449956806]
  Masked autoregressive (MAR) models unify the strengths of masked and autoregressive generation by predicting tokens in a fixed order using bidirectional attention for image generation.<n>While effective, MAR models suffer from significant computational overhead, as they recompute attention and feed-forward representations for all tokens at every decoding step.<n>We propose a training-free generation framework MARch'e to address this inefficiency through two key components: cache-aware attention and selective KV refresh.
  arXiv  Detail & Related papers  (2025-05-22T23:26:56Z)
• DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models [28.379533608574814]
  We present DyCoke, a training-free token compression method to optimize token representation and accelerate video large language models.<n>DyCoke incorporates a plug-and-play temporal compression module to minimize temporal redundancy by merging redundant tokens across frames.<n>It ensures high-quality inference by dynamically retaining the critical tokens at each decoding step.
  arXiv  Detail & Related papers  (2024-11-22T15:55:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.