Operationalizing Document AI: A Microservice Architecture for OCR and LLM Pipelines in Production

• URL: http://arxiv.org/abs/2605.18818v1
• Date: Tue, 12 May 2026 13:07:34 GMT
• Title: Operationalizing Document AI: A Microservice Architecture for OCR and LLM Pipelines in Production
• Abstract summary: We present a microservice architecture that encapsulates pipelines of multiple models for classification, optical character recognition (OCR), and large language model structured field extraction.<n>We describe our primary design decisions, including a hybrid classification, separation of GPU-bound inference from CPU-bound orchestration, use of asynchronous processing for the many IO-bound operations in the pipeline, and an independent, horizontal scaling strategy.<n>Our goal is to provide practitioners with concrete architectural patterns for building document understanding systems that work beyond the benchmark; effectively operationalizing models in production.
• Score: 0.07083294473439815
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Academic research tends to focus on new models for document understanding creating a wide gap in the literature between model definition and running models at production scale. To close that gap, we present a microservice architecture that encapsulates pipelines of multiple models for classification, optical character recognition (OCR), and large language model structured field extraction as well as our experience running this pipeline on thousands of multi-page documents per hour. We describe our primary design decisions, including a hybrid classification, separation of GPU-bound inference from CPU-bound orchestration, use of asynchronous processing for the many IO-bound operations in the pipeline, and an independent, horizontal scaling strategy. Using batch profiling, we identified two surprising qualitative findings that shape production deployments: OCR, not language-model parsing, dominates end-to-end latency, and the system saturates at a concurrency determined by shared GPU-inference capacity rather than worker count. Our goal is to provide practitioners with concrete architectural patterns for building document understanding systems that work beyond the benchmark; effectively operationalizing models in production.

Related papers

• OpenCompass: A Universal Evaluation Platform for Large Language Models [62.59671563145442]
  General-purpose large language models (LLMs) have become a critical link in advancing technological development.<n>Mainstream benchmark datasets face challenges such as the diversity of task types, inconsistent evaluation criteria, and fragmentation of data and processing.<n>This paper proposes and open-source Open LLMs, a one-stop, scalable, and high-concurrency general-purpose evaluation platform.
  arXiv  Detail & Related papers  (2026-05-19T02:50:11Z)
• DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding [63.257540233507626]
  We propose a paradigm that requires the model to execute a structured Analysis, Localization and Reasoning workflow.<n>We show it robustly generalizes from short-page training to ultra-long documents and is naturally synergistic with visual Retrieval-Augmented Generation systems.
  arXiv  Detail & Related papers  (2026-04-14T14:39:26Z)
• Leveraging Generative Models for Real-Time Query-Driven Text Summarization in Large-Scale Web Search [54.987957691350665]
  Query-Driven Text Summarization (QDTS) aims to generate concise and informative summaries from textual documents based on a given query.<n>Traditional extractive summarization models, based primarily on ranking candidate summary segments, have been the dominant approach in industrial applications.<n>We propose a novel framework to pioneer the application of generative models to address real-time QDTS in industrial web search.
  arXiv  Detail & Related papers  (2025-08-28T08:51:51Z)
• Leveraging Machine Learning and Enhanced Parallelism Detection for BPMN Model Generation from Text [75.77648333476776]
  This paper introduces an automated pipeline for extracting BPMN models from text.<n>A key contribution of this work is the introduction of a newly annotated dataset.<n>We augment the dataset with 15 newly annotated documents containing 32 parallel gateways for model training.
  arXiv  Detail & Related papers  (2025-07-11T07:25:55Z)
• OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models [58.45517851437422]
  Visually-situated text parsing (VsTP) has recently seen notable advancements, driven by the growing demand for automated document understanding.<n>Existing solutions often rely on task-specific architectures and objectives for individual tasks.<n>In this paper, we introduce Omni V2, a universal model that unifies VsTP typical tasks, including text spotting, key information extraction, table recognition, and layout analysis.
  arXiv  Detail & Related papers  (2025-02-22T09:32:01Z)
• Large Language Model Based Multi-Agent System Augmented Complex Event Processing Pipeline for Internet of Multimedia Things [8.729059187561761]
  This paper presents the development and evaluation of a Large Language Model (LLM) based system framework for complex event processing (CEP)<n>The primary goal is to create a proof-of-concept that integrates state-of-the-art LLM orchestration frameworks with publish/subscribe (pub/sub) tools to address the integration of LLMs with current CEP systems.
  arXiv  Detail & Related papers  (2025-01-01T17:38:40Z)
• Transformer Architecture for NetsDB [0.0]
  We create an end-to-end implementation of a transformer for deep learning model serving in NetsDB. We load out weights from our model for distributed processing, deployment, and efficient inferencing.
  arXiv  Detail & Related papers  (2024-05-08T04:38:36Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• Responsive parallelized architecture for deploying deep learning models in production environments [0.10152838128195467]
  Recruiters can easily shortlist candidates for jobs via viewing their curriculum vitae (CV) document. Unstructured document CV beholds candidate's portfolio and named entities listing details. The main aim of this study is to design and propose a web oriented, highly responsive, computational pipeline that systematically predicts CV entities.
  arXiv  Detail & Related papers  (2021-12-15T04:22:56Z)
• VAULT: VAriable Unified Long Text Representation for Machine Reading Comprehension [31.639069657951747]
  Existing models on Machine Reading require complex model architecture for modeling long texts with paragraph representation and classification. We propose VAULT: a light-weight and parallel-efficient paragraph representation for MRC based on contextualized representation from long document input.
  arXiv  Detail & Related papers  (2021-05-07T13:03:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.