news-crawler-LM: A Small Long-Context Model For High-Quality News Crawling

• URL: http://arxiv.org/abs/2607.21284v1
• Date: Thu, 23 Jul 2026 13:05:46 GMT
• Title: news-crawler-LM: A Small Long-Context Model For High-Quality News Crawling
• Abstract summary: Rule-based news crawlers can achieve high extraction accuracy by encoding site-specific structure.<n>Large language models provide a more flexible alternative by reducing the need for handcrafted rules.<n>We introduce news-crawler-LM, a small long-context model fine-tuned on high-quality, human-validated extractions from the Fundus news-crawling library.
• Score: 9.567186622710564
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Extracting structured content from news pages remains challenging due to heterogeneous HTML layouts, inconsistent markup, and substantial boilerplate such as navigation elements and advertisements. Rule-based news crawlers can achieve high extraction accuracy by encoding site-specific structure, but require manual configuration in order to generalize to new publishers. Large language models provide a more flexible alternative by reducing the need for handcrafted rules, but their high computational cost limits practical deployment. In this paper, we introduce news-crawler-LM, a small long-context language model fine-tuned on high-quality, human-validated extractions from the Fundus news-crawling library. Our model converts raw HTML into plaintext and structured JSON, including fields such as headline, author, publication date, and article body. In our experiments, news-crawler-LM outperforms strong baselines in HTML-to-Markdown and HTML-to-JSON extraction, improving performance by +4.8 BLEU and +6.1 METEOR in the HTML-to-Markdown task, and by +2.2 BLEU and +4.1 METEOR in the HTML-to-JSON task. However, we also observe that our model only slightly better compared to other rule-based parsing libraries on the HTML-to-plaintext task in evaluations on previously unseen publishers. We release all models and artifacts to the research community.

Related papers

• Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining [78.36592534300839]
  We show that for structured content such as tables and code blocks, extractor choice can significantly impact downstream task performance.<n>This suggests a simple intervention: by taking a Union over different extractors, we can increase the token yield of DCLM-Baseline by up to 71%.
  arXiv  Detail & Related papers  (2026-02-23T06:41:57Z)
• Dripper: Token-Efficient Main HTML Extraction with a Lightweight LM [35.10225876152952]
  We present Dripper, an efficient HTML main content extraction framework powered by lightweight language models.<n>We introduce a controlled decoding mechanism that strictly constrains the output space through logits processors.<n>Dripper achieves state-of-the-art performance across all evaluation benchmarks and outperforms all baseline methods.
  arXiv  Detail & Related papers  (2025-11-28T12:04:46Z)
• AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser [54.623900859999424]
  We introduce MinerU-HTML, a novel extraction pipeline that reformulates content extraction as a sequence labeling problem.<n>On MainWebBench, our benchmark of 7,887 annotated web pages, MinerU-HTML 81.8% ROUGE-N F1 compared to Trafilatura's 63.6%, with exceptional structured element preservation.
  arXiv  Detail & Related papers  (2025-11-20T14:15:23Z)
• Multilingual Attribute Extraction from News Web Pages [44.99833362998488]
  This paper addresses the challenge of automatically extracting attributes from news article web pages across multiple languages.<n>We prepared a multilingual dataset comprising 3,172 marked-up news web pages across six languages (English, German, Russian, Chinese, Korean, and Arabic).<n>We fine-tuned the pre-trained state-of-the-art model, MarkupLM, to extract news attributes from these pages and evaluated the impact of translating pages into English on extraction quality.
  arXiv  Detail & Related papers  (2025-02-04T09:43:40Z)
• HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems [62.36019283532854]
  Retrieval-Augmented Generation (RAG) has been shown to improve knowledge capabilities and alleviate the hallucination problem of LLMs.<n>We propose RAG, which uses HTML instead of plain text as the format of retrieved knowledge in RAG.
  arXiv  Detail & Related papers  (2024-11-05T09:58:36Z)
• AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation [54.17246674188208]
  Web scraping is a powerful technique that extracts data from websites, enabling automated data collection, enhancing data analysis capabilities, and minimizing manual data entry efforts. Existing methods, wrappers-based methods suffer from limited adaptability and scalability when faced with a new website. We introduce the paradigm of generating web scrapers with large language models (LLMs) and propose AutoScraper, a two-stage framework that can handle diverse and changing web environments more efficiently.
  arXiv  Detail & Related papers  (2024-04-19T09:59:44Z)
• Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions [2.2863439039616127]
  Fundus is a user-friendly news scraper that enables users to obtain millions of high-quality news articles with just a few lines of code. Unlike existing news scrapers, we use manually crafted, bespoke content extractors. Our framework combines both crawling (retrieving HTML from the web or large web archives) and content extraction into a single pipeline.
  arXiv  Detail & Related papers  (2024-03-22T15:22:06Z)
• Understanding HTML with Large Language Models [73.92747433749271]
  Large language models (LLMs) have shown exceptional performance on a variety of natural language tasks. We contribute HTML understanding models (fine-tuned LLMs) and an in-depth analysis of their capabilities under three tasks. We show that LLMs pretrained on standard natural language corpora transfer remarkably well to HTML understanding tasks.
  arXiv  Detail & Related papers  (2022-10-08T07:27:17Z)
• HTLM: Hyper-Text Pre-Training and Prompting of Language Models [52.32659647159799]
  We introduce HTLM, a hyper-text language model trained on a large-scale web crawl. We show that pretraining with a BART-style denoising loss directly on simplified HTML provides highly effective transfer for a wide range of end tasks and supervision levels. We find that hyper-text prompts provide more value to HTLM, in terms of data efficiency, than plain text prompts do for existing LMs.
  arXiv  Detail & Related papers  (2021-07-14T19:39:31Z)
• Boilerplate Removal using a Neural Sequence Labeling Model [4.056234173482691]
  We propose a neural sequence labeling model that does not rely on any hand-crafted features but takes only the HTML tags and words that appear in a web page as input. This allows us to present a browser extension which highlights the content of arbitrary web pages directly within the browser using our model.
  arXiv  Detail & Related papers  (2020-04-22T08:06:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.