Institutional Books - Enriched Text: A customizable multilingual open-source pipeline for denoising, deduplicating, and annotating OCR text at scale

• URL: http://arxiv.org/abs/2608.19026v1
• Date: Wed, 19 Aug 2026 15:20:54 GMT
• Title: Institutional Books - Enriched Text: A customizable multilingual open-source pipeline for denoising, deduplicating, and annotating OCR text at scale
• Abstract summary: We describe an approach that we call Enriched Text.<n>Instead of producing a single 'complete' stream of tokens, we normalize the text while preserving metadata through annotations.<n>We provide this information through HTML-like annotations layered on top of the text.
• Score: 5.31661802351755
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Released in 2025, Institutional Books: Harvard Library (IB-HL) is a collection of 983,004 volumes (242B o200k_base tokens), originally digitized through Harvard Library's participation in the Google Books Library project. As researchers and developers have begun to use IB-HL, a tension has emerged between standard large-scale preprocessing practices and the goals of careful information stewardship. Many existing pipelines optimize for web text: as a result, they tend to aggressively filter, deduplicate, restrict by language, and sometimes discard meaningful metadata. Meanwhile, researchers seeking to use IB-HL duplicate effort while performing similar processing and analysis. We describe an approach that we call Enriched Text. Instead of producing a single 'complete' stream of tokens, we normalize the text while preserving metadata through annotations. We separate endmatter, detect per-paragraph language, identify clusters of duplicate paragraphs, and compute per-paragraph bits-per-byte scores. We provide this information through HTML-like annotations layered on top of the text. By parsing these annotations, users can tailor the output to their own needs instead of accepting a global editorial decision on content. The pipeline applies to all $\approx$250 languages in the collection. This report describes this project's goals, implementation, and design rationale. The release includes IB-HL-ET (an enriched-text version of IB-HL containing 217B o200k_base tokens across 983,003 volumes, organized into 1.39B annotated subtopic paragraphs) and the pipeline that produced it. These serve to make the collection easier for machines to parse and for humans to study.

Related papers

• UniRec-0.1B: Unified Text and Formula Recognition with 0.1B Parameters [55.34921520578968]
  Vision-language models (VLMs) have achieved impressive unified recognition of text and formulas.<n>We propose UniRec-0.1B, a unified recognition model with only 0.1B parameters.<n>It is capable of performing text and formula recognition at multiple levels, including characters, words, lines, paragraphs, and documents.
  arXiv  Detail & Related papers  (2025-12-24T10:35:21Z)
• HPLT 3.0: Very Large-Scale Multilingual Resources for LLM and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models [25.953042884928006]
  We present an initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages.<n>At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data.<n>We train and evaluate a family of 57 monolingual encoder-decoder models, as well as a handful of monolingual GPT-like reference models.
  arXiv  Detail & Related papers  (2025-11-02T20:16:38Z)
• FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language [48.79534869177174]
  We introduce a new pre-training dataset curation pipeline based on FineWeb.<n>We show that our pipeline can be used to create non-English corpora that produce more performant models than prior datasets.<n>We scale our pipeline to over 1000 languages using almost 100 Common Crawl snapshots to produce FineWeb2, a new 20 terabyte (5 billion document) multilingual dataset.
  arXiv  Detail & Related papers  (2025-06-26T01:01:47Z)
• Semantic Outlier Removal with Embedding Models and LLMs [0.45080838507508303]
  We introduce SORE (Semantic Outlier Removal), a cost-effective, transparent method to identify and excise unwanted text segments.<n>SORE achieves near-LLM extraction precision at a fraction of the cost.<n>Our system is currently deployed in production, processing millions of documents daily across multiple languages.
  arXiv  Detail & Related papers  (2025-06-19T23:06:12Z)
• Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usability [1.3281177137699656]
  This report introduces Institutional Books 1.0, a collection of public domain books originally digitized through Harvard Library's participation in the Google Books project, beginning in 2006.<n>Working with Harvard Library, we extracted, analyzed, and processed these volumes into an extensively-documented dataset of historic texts.<n>This analysis covers the entirety of Harvard Library's collection scanned as part of that project, originally spanning 1,075,899 volumes written in over 250 different languages for a total of approximately 250 billion tokens.
  arXiv  Detail & Related papers  (2025-06-10T00:11:30Z)
• Low-Resource Language Processing: An OCR-Driven Summarization and Translation Pipeline [0.0]
  This paper presents an end-to-end suite for multilingual information extraction and processing from image-based documents.<n>The system uses Optical Character Recognition (Tesseract) to extract text in languages such as English, Hindi, and Tamil.<n>The current research shows a real-world application of libraries, models, and APIs to close the language gap and enhance access to information in image media across different linguistic environments.
  arXiv  Detail & Related papers  (2025-05-16T12:20:37Z)
• Discourse Features Enhance Detection of Document-Level Machine-Generated Content [53.41994768824785]
  Machine-generated content poses challenges such as academic plagiarism and the spread of misinformation.<n>Existing MGC detectors often focus solely on surface-level information, overlooking implicit and structural features.<n>We introduce novel methodologies and datasets to overcome these challenges.
  arXiv  Detail & Related papers  (2024-12-17T08:47:41Z)
• MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions [54.08017526771947]
  Multilingual Reverse Instructions (MURI) generates high-quality instruction tuning datasets for low-resource languages. MURI produces instruction-output pairs from existing human-written texts in low-resource languages. Our dataset, MURI-IT, includes more than 2 million instruction-output pairs across 200 languages.
  arXiv  Detail & Related papers  (2024-09-19T17:59:20Z)
• Batching BPE Tokenization Merges [55.2480439325792]
  BatchBPE is an open-source pure Python implementation of the Byte Pair algorithm. It is used to train a high quality tokenizer on a basic laptop.
  arXiv  Detail & Related papers  (2024-08-05T09:37:21Z)
• LexMatcher: Dictionary-centric Data Collection for LLM-based Machine Translation [67.24113079928668]
  We present LexMatcher, a method for data curation driven by the coverage of senses found in bilingual dictionaries. Our approach outperforms the established baselines on the WMT2022 test sets.
  arXiv  Detail & Related papers  (2024-06-03T15:30:36Z)
• Evaluating Embedding APIs for Information Retrieval [51.24236853841468]
  We evaluate the capabilities of existing semantic embedding APIs on domain generalization and multilingual retrieval. We find that re-ranking BM25 results using the APIs is a budget-friendly approach and is most effective in English. For non-English retrieval, re-ranking still improves the results, but a hybrid model with BM25 works best, albeit at a higher cost.
  arXiv  Detail & Related papers  (2023-05-10T16:40:52Z)
• Documenting the English Colossal Clean Crawled Corpus [28.008953329187648]
  This work provides the first documentation for the Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set of filters to a single snapshot of Common Crawl. We begin with a high-level summary of the data, including distributions of where the text came from and when it was written. We then give more detailed analysis on salient parts of this data, including the most frequent sources of text.
  arXiv  Detail & Related papers  (2021-04-18T07:42:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.