Identify, Locate, Link: End-to-End Key-Value Extraction from Document Images

• URL: http://arxiv.org/abs/2608.20868v1
• Date: Fri, 21 Aug 2026 08:36:23 GMT
• Title: Identify, Locate, Link: End-to-End Key-Value Extraction from Document Images
• Abstract summary: Document processing pipelines traditionally cascade optical character recognition (OCR) engines with downstream models for structured information extraction.<n>We fine-tune SmolDocling, a compact 256M- parameter vision-language model (VLM), to perform end-to-end key-value extraction directly from document images.<n>We extend DocTags with specialized key, value, region, and link tags, enabling many-to-many relationships in a unified output sequence.
• Score: 4.971112189386969
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Document processing pipelines traditionally cascade optical character recognition (OCR) engines with downstream models for structured information extraction, leading to multi-stage error propagation. We fine-tune SmolDocling, a compact 256M-parameter vision-language model (VLM), to perform end-to-end key-value extraction directly from document images, jointly solving identification, localization, and association in a single pass without OCR preprocessing. We extend DocTags with specialized key, value, region, and link tags, enabling many-to-many relationships in a unified output sequence. To address data limitations, we design an augmentation pipeline combining synthetic form filling and graph-based crops that preserve complete key-value subgraphs. We further introduce a layout-aware evaluation framework extending text matching with spatial bounding box verification. On FUNSD, XFUND, and a large-scale private dataset, our model outperforms larger zero-shot VLM baselines under layout-aware evaluation, while being 27 times smaller than Qwen2.5-VL (7B) and over 5 times faster at inference. The model weights will be released publicly after publication.

Related papers

• MonkeyOCR v1.5 Technical Report: Unlocking Robust Document Parsing for Complex Patterns [80.05126590825121]
  MonkeyOCR v1.5 is a unified vision-language framework that enhances both layout understanding and content recognition.<n>To address complex table structures, we propose a visual consistency-based reinforcement learning scheme.<n>Two specialized modules, Image-Decoupled Table Parsing and Type-Guided Table Merging, are introduced to enable reliable parsing of tables.
  arXiv  Detail & Related papers  (2025-11-13T15:12:17Z)
• ModernVBERT: Towards Smaller Visual Document Retrievers [8.752477008109844]
  ModernVBERT is a compact vision-language encoder that outperforms models up to 10 times larger when finetuned on document retrieval tasks.<n>We measure the impact of attention masking, image resolution, modality alignment data regimes, and late interaction centered contrastive objectives which emerge as central performance factors.
  arXiv  Detail & Related papers  (2025-10-01T17:41:17Z)
• VDInstruct: Zero-Shot Key Information Extraction via Content-Aware Vision Tokenization [7.769156392417315]
  Key Information Extraction underpins the understanding of visual documents.<n>Existing multimodal large language models (MLLMs) often perform poorly on dense documents.<n>We introduce VDInstruct, an MLLM that separates spatial region detection from semantic feature extraction.
  arXiv  Detail & Related papers  (2025-07-13T08:15:11Z)
• Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing [46.14775667559124]
  layoutRL is an end-to-end reinforcement learning framework that trains models to be explicitly layout-aware.<n>We will publicly release our code and dataset to accelerate progress in robust document understanding.
  arXiv  Detail & Related papers  (2025-06-01T15:19:52Z)
• Towards Text-Image Interleaved Retrieval [49.96332254241075]
  We introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences.<n>We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries.<n>We propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity.
  arXiv  Detail & Related papers  (2025-02-18T12:00:47Z)
• TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document [60.01330653769726]
  We present TextMonkey, a large multimodal model (LMM) tailored for text-centric tasks. By adopting Shifted Window Attention with zero-initialization, we achieve cross-window connectivity at higher input resolutions. By expanding its capabilities to encompass text spotting and grounding, and incorporating positional information into responses, we enhance interpretability.
  arXiv  Detail & Related papers  (2024-03-07T13:16:24Z)
• LOCOST: State-Space Models for Long Document Abstractive Summarization [76.31514220737272]
  We propose LOCOST: an encoder-decoder architecture based on state-space models for conditional text generation with long context inputs. With a computational complexity of $O(L log L)$, this architecture can handle significantly longer sequences than state-of-the-art models that are based on sparse attention patterns.
  arXiv  Detail & Related papers  (2024-01-31T15:33:37Z)
• One-shot Key Information Extraction from Document with Deep Partial Graph Matching [60.48651298832829]
  Key Information Extraction (KIE) from documents improves efficiency, productivity, and security in many industrial scenarios. Existing supervised learning methods for the KIE task need to feed a large number of labeled samples and learn separate models for different types of documents. We propose a deep end-to-end trainable network for one-shot KIE using partial graph matching.
  arXiv  Detail & Related papers  (2021-09-26T07:45:53Z)
• Spatial Dual-Modality Graph Reasoning for Key Information Extraction [31.04597531115209]
  We propose an end-to-end Spatial Dual-Modality Graph Reasoning method (SDMG-R) to extract key information from unstructured document images. We release a new dataset named WildReceipt, which is collected and annotated for the evaluation of key information extraction from document images of unseen templates in the wild.
  arXiv  Detail & Related papers  (2021-03-26T13:46:00Z)
• Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching [28.190001111358438]
  We propose a Siamese Multi-depth Transformer-based SMITH for long-form document matching. Our model contains several innovations to adapt self-attention models for longer text input. We will open source a Wikipedia based benchmark dataset, code and a pre-trained checkpoint to accelerate future research on long-form document matching.
  arXiv  Detail & Related papers  (2020-04-26T07:04:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.