Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions

• URL: http://arxiv.org/abs/2403.15279v2
• Date: Mon, 24 Jun 2024 09:18:48 GMT
• Title: Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions
• Abstract summary: Fundus is a user-friendly news scraper that enables users to obtain millions of high-quality news articles with just a few lines of code. Unlike existing news scrapers, we use manually crafted, bespoke content extractors. Our framework combines both crawling (retrieving HTML from the web or large web archives) and content extraction into a single pipeline.
• Score: 2.2863439039616127
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces Fundus, a user-friendly news scraper that enables users to obtain millions of high-quality news articles with just a few lines of code. Unlike existing news scrapers, we use manually crafted, bespoke content extractors that are specifically tailored to the formatting guidelines of each supported online newspaper. This allows us to optimize our scraping for quality such that retrieved news articles are textually complete and without HTML artifacts. Further, our framework combines both crawling (retrieving HTML from the web or large web archives) and content extraction into a single pipeline. By providing a unified interface for a predefined collection of newspapers, we aim to make Fundus broadly usable even for non-technical users. This paper gives an overview of the framework, discusses our design choices, and presents a comparative evaluation against other popular news scrapers. Our evaluation shows that Fundus yields significantly higher quality extractions (complete and artifact-free news articles) than prior work. The framework is available on GitHub under https://github.com/flairNLP/fundus and can be simply installed using pip.

Related papers

• news-crawler-LM: A Small Long-Context Model For High-Quality News Crawling [9.567186622710564]
  Rule-based news crawlers can achieve high extraction accuracy by encoding site-specific structure.<n>Large language models provide a more flexible alternative by reducing the need for handcrafted rules.<n>We introduce news-crawler-LM, a small long-context model fine-tuned on high-quality, human-validated extractions from the Fundus news-crawling library.
  arXiv  Detail & Related papers  (2026-07-23T13:05:46Z)
• Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining [78.36592534300839]
  We show that for structured content such as tables and code blocks, extractor choice can significantly impact downstream task performance.<n>This suggests a simple intervention: by taking a Union over different extractors, we can increase the token yield of DCLM-Baseline by up to 71%.
  arXiv  Detail & Related papers  (2026-02-23T06:41:57Z)
• AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser [54.623900859999424]
  We introduce MinerU-HTML, a novel extraction pipeline that reformulates content extraction as a sequence labeling problem.<n>On MainWebBench, our benchmark of 7,887 annotated web pages, MinerU-HTML 81.8% ROUGE-N F1 compared to Trafilatura's 63.6%, with exceptional structured element preservation.
  arXiv  Detail & Related papers  (2025-11-20T14:15:23Z)
• Multi-view autoencoders for Fake News Detection [5.863538874435322]
  This paper proposes using multi-view autoencoders to generate a joint feature representation for fake news detection.<n>Experiments on fake news datasets show a significant improvement in classification performance compared to individual views.
  arXiv  Detail & Related papers  (2025-04-10T19:59:34Z)
• AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation [54.17246674188208]
  Web scraping is a powerful technique that extracts data from websites, enabling automated data collection, enhancing data analysis capabilities, and minimizing manual data entry efforts. Existing methods, wrappers-based methods suffer from limited adaptability and scalability when faced with a new website. We introduce the paradigm of generating web scrapers with large language models (LLMs) and propose AutoScraper, a two-stage framework that can handle diverse and changing web environments more efficiently.
  arXiv  Detail & Related papers  (2024-04-19T09:59:44Z)
• FineFake: A Knowledge-Enriched Dataset for Fine-Grained Multi-Domain Fake News Detection [54.37159298632628]
  FineFake is a multi-domain knowledge-enhanced benchmark for fake news detection. FineFake encompasses 16,909 data samples spanning six semantic topics and eight platforms. The entire FineFake project is publicly accessible as an open-source repository.
  arXiv  Detail & Related papers  (2024-03-30T14:39:09Z)
• Cleaner Pretraining Corpus Curation with Neural Web Scraping [39.97459187762505]
  This paper presents a simple, fast, and effective Neural web Scraper (NeuScraper) to help extract primary and clean text contents from webpages. Experimental results show that NeuScraper surpasses the baseline scrapers by achieving more than a 20% improvement.
  arXiv  Detail & Related papers  (2024-02-22T16:04:03Z)
• Focus! Relevant and Sufficient Context Selection for News Image Captioning [69.36678144800936]
  News Image Captioning requires describing an image by leveraging additional context from a news article. We propose to use the pre-trained vision and language retrieval model CLIP to localize the visually grounded entities in the news article. Our experiments demonstrate that by simply selecting a better context from the article, we can significantly improve the performance of existing models.
  arXiv  Detail & Related papers  (2022-12-01T20:00:27Z)
• TRIE++: Towards End-to-End Information Extraction from Visually Rich Documents [51.744527199305445]
  This paper proposes a unified end-to-end information extraction framework from visually rich documents. Text reading and information extraction can reinforce each other via a well-designed multi-modal context block. The framework can be trained in an end-to-end trainable manner, achieving global optimization.
  arXiv  Detail & Related papers  (2022-07-14T08:52:07Z)
• Documenting the English Colossal Clean Crawled Corpus [28.008953329187648]
  This work provides the first documentation for the Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set of filters to a single snapshot of Common Crawl. We begin with a high-level summary of the data, including distributions of where the text came from and when it was written. We then give more detailed analysis on salient parts of this data, including the most frequent sources of text.
  arXiv  Detail & Related papers  (2021-04-18T07:42:52Z)
• docExtractor: An off-the-shelf historical document element extraction [18.828438308738495]
  We present docExtractor, a generic approach for extracting visual elements such as text lines or illustrations from historical documents. We demonstrate it provides high-quality performances as an off-the-shelf system across a wide variety of datasets. We introduce a new public dataset dubbed IlluHisDoc dedicated to the fine evaluation of illustration segmentation in historical documents.
  arXiv  Detail & Related papers  (2020-12-15T10:19:18Z)
• Graph-based Topic Extraction from Vector Embeddings of Text Documents: Application to a Corpus of News Articles [0.0]
  We present an unsupervised framework that brings together powerful vector embeddings from natural language processing with tools from multiscale graph partitioning. We show the advantages of graph-based clustering through end-to-end comparisons with other popular clustering and topic modelling methods. This work is showcased through an analysis of a corpus of US news coverage during the presidential election year of 2016.
  arXiv  Detail & Related papers  (2020-10-28T16:20:05Z)
• Context-Based Quotation Recommendation [60.93257124507105]
  We propose a novel context-aware quote recommendation system. It generates a ranked list of quotable paragraphs and spans of tokens from a given source document. We conduct experiments on a collection of speech transcripts and associated news articles.
  arXiv  Detail & Related papers  (2020-05-17T17:49:53Z)
• ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpages [66.45377533562417]
  We propose a solution for "zero-shot" open-domain relation extraction from webpages with a previously unseen template. Our model uses a graph neural network-based approach to build a rich representation of text fields on a webpage.
  arXiv  Detail & Related papers  (2020-05-14T16:15:58Z)
• BaitWatcher: A lightweight web interface for the detection of incongruent news headlines [27.29585619643952]
  BaitWatcher is a lightweight web interface that guides readers in estimating the likelihood of incongruence in news articles before clicking on the headlines. BaiittWatcher utilizes a hierarchical recurrent encoder that efficiently learns complex textual representations of a news headline and its associated body text.
  arXiv  Detail & Related papers  (2020-03-23T23:43:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.