Institutional Books - Visual Elements: An open-source pipeline for extracting, classifying, deduplicating, and captioning visual elements from digital book collections

• URL: http://arxiv.org/abs/2608.18957v1
• Date: Wed, 19 Aug 2026 14:21:56 GMT
• Title: Institutional Books - Visual Elements: An open-source pipeline for extracting, classifying, deduplicating, and captioning visual elements from digital book collections
• Abstract summary: This report introduces Institutional Books - Visual Elements, an open-source pipeline for detecting, classifying, deduplicating, and captioning visual elements from historical book collections.<n>We release an initial dataset of 22.6 million visual elements extracted from the 983,004 scanned volumes that comprise the Institutional Books: Harvard Library dataset.
• Score: 5.123928181133507
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Historical book collections contain rich visual elements - such as illustrations, photographs, engravings, and decorative art - that are frequently under-explored in large-scale digitization projects. While Optical Character Recognition (OCR) has standardized the extraction of textual content, these visual components offer a layer of nuance and context that remains largely untapped by automated text extraction workflows. This technical report introduces Institutional Books - Visual Elements, an open-source end-to-end pipeline for detecting, classifying, deduplicating, and captioning visual elements from historical book collections. Alongside this pipeline, we release an initial dataset of 22.6 million visual elements extracted from the 983,004 scanned volumes that comprise the Institutional Books: Harvard Library dataset. This work contributes to ongoing, community-wide efforts to enable new use cases for digitized library collections through computational access, from artificial intelligence model training to digital humanities research.

Related papers

• Multimodal OCR: Parse Anything from Documents [72.69545534962234]
  dots.mocr treats visual elements such as charts, diagrams, tables, and icons as first-class parsing targets.<n>It reconstructs both text and graphics as structured outputs, enabling more faithful document reconstruction.<n>It supports end-to-end training over heterogeneous document elements.
  arXiv  Detail & Related papers  (2026-03-13T14:42:21Z)
• Knowledge Graphs for Digitized Manuscripts in Jagiellonian Digital Library Application [8.732274235941974]
  Galleries, libraries, archives and museums (GLAM institutions) are actively digitizing their holdings and creates extensive digital collections.<n>These collections are often enriched with metadata describing items but not exactly their contents.<n>We explore an integrated methodology of computer vision (CV), artificial intelligence (AI) and semantic web technologies to enrich metadata and construct knowledge graphs for digitized manuscripts and incunabula.
  arXiv  Detail & Related papers  (2025-05-29T14:49:24Z)
• Generative Compositor for Few-Shot Visual Information Extraction [60.663887314625164]
  We propose a novel generative model, named Generative generative spatialtor, to address the challenge of few-shot VIE.<n>Generative generative spatialtor is a hybrid pointer-generator network that emulates the operations of a compositor by retrieving words from the source text.<n>The proposed method achieves highly competitive results in the full-sample training, while notably outperforms the baseline in the 1-shot, 5-shot, and 10-shot settings.
  arXiv  Detail & Related papers  (2025-03-21T04:56:24Z)
• Making History Readable [0.0]
  This poster highlights three collections focusing on handwritten letters, newspapers, and digitized topographic maps. We discuss the challenges with each collection and detail our approaches to address them. Our proposed methods aim to enhance the user experience by making the contents in these collections easier to search and navigate.
  arXiv  Detail & Related papers  (2024-11-26T17:06:58Z)
• A Library Perspective on Supervised Text Processing in Digital Libraries: An Investigation in the Biomedical Domain [3.9519587827662397]
  We focus on relation extraction and text classification, using the showcase of eight biomedical benchmarks. We consider trade-offs between accuracy and application costs, dive into training data generation through distant supervision and large language models such as ChatGPT, LLama, and Olmo, and discuss how to design final pipelines.
  arXiv  Detail & Related papers  (2024-11-06T07:54:10Z)
• Visual Navigation of Digital Libraries: Retrieval and Classification of Images in the National Library of Norway's Digitised Book Collection [0.3277163122167433]
  We present a proof-of-concept image search application for exploring images in the National Library of Norway's pre-1900 books. We compare Vision Transformer (ViT), Contrastive Language-Image Pre-training (CLIP), and Sigmoid loss for Language-Image Pre-training (SigLIP) embeddings for image retrieval and classification.
  arXiv  Detail & Related papers  (2024-10-19T04:20:23Z)
• Constructing Image-Text Pair Dataset from Books [10.92677060085447]
  We propose a novel approach to leverage digital archives for machine learning. In our experiments, we apply our pipeline on old photo books to construct an image-text pair dataset.
  arXiv  Detail & Related papers  (2023-10-03T10:23:28Z)
• DocumentNet: Bridging the Data Gap in Document Pre-Training [78.01647768018485]
  We propose a method to collect massive-scale and weakly labeled data from the web to benefit the training of VDER models. The collected dataset, named DocumentNet, does not depend on specific document types or entity sets. Experiments on a set of broadly adopted VDER tasks show significant improvements when DocumentNet is incorporated into the pre-training.
  arXiv  Detail & Related papers  (2023-06-15T08:21:15Z)
• TRIE++: Towards End-to-End Information Extraction from Visually Rich Documents [51.744527199305445]
  This paper proposes a unified end-to-end information extraction framework from visually rich documents. Text reading and information extraction can reinforce each other via a well-designed multi-modal context block. The framework can be trained in an end-to-end trainable manner, achieving global optimization.
  arXiv  Detail & Related papers  (2022-07-14T08:52:07Z)
• Layout-Aware Information Extraction for Document-Grounded Dialogue: Dataset, Method and Demonstration [75.47708732473586]
  We propose a layout-aware document-level Information Extraction dataset, LIE, to facilitate the study of extracting both structural and semantic knowledge from visually rich documents. LIE contains 62k annotations of three extraction tasks from 4,061 pages in product and official documents. Empirical results show that layout is critical for VRD-based extraction, and system demonstration also verifies that the extracted knowledge can help locate the answers that users care about.
  arXiv  Detail & Related papers  (2022-07-14T07:59:45Z)
• Digital Editions as Distant Supervision for Layout Analysis of Printed Books [76.29918490722902]
  We describe methods for exploiting this semantic markup as distant supervision for training and evaluating layout analysis models. In experiments with several model architectures on the half-million pages of the Deutsches Textarchiv (DTA), we find a high correlation of these region-level evaluation methods with pixel-level and word-level metrics. We discuss the possibilities for improving accuracy with self-training and the ability of models trained on the DTA to generalize to other historical printed books.
  arXiv  Detail & Related papers  (2021-12-23T16:51:53Z)
• Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph [96.95815946327079]
  It is difficult to learn the association between named entities and visual cues due to the long-tail distribution of named entities. We propose a novel approach that constructs a multi-modal knowledge graph to associate the visual objects with named entities.
  arXiv  Detail & Related papers  (2021-07-26T05:50:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.