ReaderLM-v2: Small Language Model for HTML to Markdown and JSON

• URL: http://arxiv.org/abs/2503.01151v1
• Date: Mon, 03 Mar 2025 03:57:04 GMT
• Title: ReaderLM-v2: Small Language Model for HTML to Markdown and JSON
• Authors: Feng Wang, Zesheng Shi, Bo Wang, Nan Wang, Han Xiao,
• Abstract summary: We present ReaderLM-v2, a compact 1.5 billion parameter language model designed for efficient web content extraction.<n>Our model processes documents up to 512K messy HTML into clean or markdown formats with high accuracy -- making it an ideal tool for grounding large language models.
• Score: 7.9969849952515775
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present ReaderLM-v2, a compact 1.5 billion parameter language model designed for efficient web content extraction. Our model processes documents up to 512K tokens, transforming messy HTML into clean Markdown or JSON formats with high accuracy -- making it an ideal tool for grounding large language models. The model's effectiveness results from two key innovations: (1) a three-stage data synthesis pipeline that generates high quality, diverse training data by iteratively drafting, refining, and critiquing web content extraction; and (2) a unified training framework combining continuous pre-training with multi-objective optimization. Intensive evaluation demonstrates that ReaderLM-v2 outperforms GPT-4o-2024-08-06 and other larger models by 15-20\% on carefully curated benchmarks, particularly excelling at documents exceeding 100K tokens, while maintaining significantly lower computational requirements.

Related papers

• MiniCPM4: Ultra-Efficient LLMs on End Devices [124.73631357883228]
  MiniCPM4 is a highly efficient large language model (LLM) designed explicitly for end-side devices.<n>We achieve this efficiency through systematic innovation in four key dimensions: model architecture, training data, training algorithms, and inference systems.<n>MiniCPM4 is available in two versions, with 0.5B and 8B parameters, respectively.
  arXiv  Detail & Related papers  (2025-06-09T16:16:50Z)
• MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm [60.14048367611333]
  MonkeyOCR is a vision-language model for document parsing.<n>It advances the state of the art by leveraging a Structure-Recognition-Relation (SRR) triplet paradigm.
  arXiv  Detail & Related papers  (2025-06-05T16:34:57Z)
• GneissWeb: Preparing High Quality Data for LLMs at Scale [15.596915267015797]
  We introduce GneissWeb, a large dataset yielding around 10 trillion tokens.<n>GneissWeb achieves a favorable trade-off between data quality and quantity.<n>We show that models trained using GneissWeb dataset outperform those trained on FineWeb-V1.1.0.
  arXiv  Detail & Related papers  (2025-02-19T00:14:29Z)
• PDF-WuKong: A Large Multimodal Model for Efficient Long PDF Reading with End-to-End Sparse Sampling [63.93112754821312]
  Multimodal document understanding is a challenging task to process and comprehend large amounts of textual and visual information.<n>Recent advances in Large Language Models (LLMs) have significantly improved the performance of this task.<n>We introduce PDF-WuKong, a multimodal large language model (MLLM) which is designed to enhance multimodal question-answering (QA) for long PDF documents.
  arXiv  Detail & Related papers  (2024-10-08T12:17:42Z)
• DecorateLM: Data Engineering through Corpus Rating, Tagging, and Editing with Language Models [78.51470038301436]
  We introduce DecorateLM, a data engineering method designed to refine the pretraining corpus through data rating, tagging and editing. We then apply DecorateLM to enhance 100 billion tokens of the training corpus, selecting 45 billion tokens that exemplify high quality and diversity for the further training of another 1.2 billion parameter LLM. Our results demonstrate that employing such high-quality data can significantly boost model performance, showcasing a powerful approach to enhance the quality of the pretraining corpus.
  arXiv  Detail & Related papers  (2024-10-08T02:42:56Z)
• Target-Aware Language Modeling via Granular Data Sampling [25.957424920194914]
  Language model pretraining generally targets a broad range of use cases and incorporates data from diverse sources. A cost-effective and straightforward approach is sampling with low-dimensional data features. We show that pretrained models perform on par with the full RefinedWeb data and outperform randomly selected samples for model sizes ranging from 125M to 1.5B.
  arXiv  Detail & Related papers  (2024-09-23T04:52:17Z)
• Large Language Model-guided Document Selection [23.673690115025913]
  Large Language Model (LLM) pre-training exhausts an ever growing compute budget. Recent research has demonstrated that careful document selection enables comparable model quality with only a fraction of the FLOPs. We explore a promising direction for scalable general-domain document selection.
  arXiv  Detail & Related papers  (2024-06-07T04:52:46Z)
• InternLM2 Technical Report [159.70692271378581]
  This paper introduces InternLM2, an open-source Large Language Models (LLMs) that outperforms its predecessors in comprehensive evaluations across 6 dimensions and 30 benchmarks. The pre-training process of InternLM2 is meticulously detailed, highlighting the preparation of diverse data types. InternLM2 efficiently captures long-term dependencies, initially trained on 4k tokens before advancing to 32k tokens in pre-training and fine-tuning stages.
  arXiv  Detail & Related papers  (2024-03-26T00:53:24Z)
• The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only [48.498376125522114]
  We show that properly filtered and deduplicated web data alone can lead to powerful models. We release an extract of 600 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.
  arXiv  Detail & Related papers  (2023-06-01T20:03:56Z)
• Understanding HTML with Large Language Models [73.92747433749271]
  Large language models (LLMs) have shown exceptional performance on a variety of natural language tasks. We contribute HTML understanding models (fine-tuned LLMs) and an in-depth analysis of their capabilities under three tasks. We show that LLMs pretrained on standard natural language corpora transfer remarkably well to HTML understanding tasks.
  arXiv  Detail & Related papers  (2022-10-08T07:27:17Z)
• Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization [108.09419317477986]
  Z-Code++ is a new pre-trained language model optimized for abstractive text summarization. The model is first pre-trained using text corpora for language understanding, and then is continually pre-trained on summarization corpora for grounded text generation. Our model is parameter-efficient in that it outperforms the 600x larger PaLM-540B on XSum, and the finetuned 200x larger GPT3-175B on SAMSum.
  arXiv  Detail & Related papers  (2022-08-21T01:00:54Z)
• Long Document Summarization with Top-down and Bottom-up Inference [113.29319668246407]
  We propose a principled inference framework to improve summarization models on two aspects. Our framework assumes a hierarchical latent structure of a document where the top-level captures the long range dependency. We demonstrate the effectiveness of the proposed framework on a diverse set of summarization datasets.
  arXiv  Detail & Related papers  (2022-03-15T01:24:51Z)
• The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild [0.0]
  The dataset consists of various challenges related to web-based data, such as machine translated content, encoding errors, multiple contents presented in one document etc. The initial machine learning experiments on the dataset show that (1) pre-Transformer models are drastically less able to model the phenomena, with macro F1 metrics ranging around 0.22, while Transformer-based models achieve scores of around 0.58, and (2) multilingual Transformer models work as well on the task as the monolingual models that were previously proven to be superior to multilingual models on standard NLP tasks.
  arXiv  Detail & Related papers  (2022-01-11T09:39:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.