Newswire: A Large-Scale Structured Database of a Century of Historical News

• URL: http://arxiv.org/abs/2406.09490v1
• Date: Thu, 13 Jun 2024 16:20:05 GMT
• Title: Newswire: A Large-Scale Structured Database of a Century of Historical News
• Authors: Emily Silcock, Abhishek Arora, Luca D'Amico-Wong, Melissa Dell,
• Abstract summary: Historians argue that newswires played a pivotal role in creating a national identity and shared understanding of the world. We reconstruct such an archive by applying a customized deep learning pipeline to hundreds of terabytes of raw image scans from thousands of local newspapers. The resulting dataset contains 2.7 million unique public domain U.S. newswire articles, written between 1878 and 1977.
• Score: 3.562368079040469
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the U.S. historically, local newspapers drew their content largely from newswires like the Associated Press. Historians argue that newswires played a pivotal role in creating a national identity and shared understanding of the world, but there is no comprehensive archive of the content sent over newswires. We reconstruct such an archive by applying a customized deep learning pipeline to hundreds of terabytes of raw image scans from thousands of local newspapers. The resulting dataset contains 2.7 million unique public domain U.S. newswire articles, written between 1878 and 1977. Locations in these articles are georeferenced, topics are tagged using customized neural topic classification, named entities are recognized, and individuals are disambiguated to Wikipedia using a novel entity disambiguation model. To construct the Newswire dataset, we first recognize newspaper layouts and transcribe around 138 millions structured article texts from raw image scans. We then use a customized neural bi-encoder model to de-duplicate reproduced articles, in the presence of considerable abridgement and noise, quantifying how widely each article was reproduced. A text classifier is used to ensure that we only include newswire articles, which historically are in the public domain. The structured data that accompany the texts provide rich information about the who (disambiguated individuals), what (topics), and where (georeferencing) of the news that millions of Americans read over the course of a century. We also include Library of Congress metadata information about the newspapers that ran the articles on their front pages. The Newswire dataset is useful both for large language modeling - expanding training data beyond what is available from modern web texts - and for studying a diversity of questions in computational linguistics, social science, and the digital humanities.

Related papers

• 3DLNews: A Three-decade Dataset of US Local News Articles [49.1574468325115]
  3DLNews is a novel dataset with local news articles from the United States spanning the period from 1996 to 2024. It contains almost 1 million URLs (with HTML text) from over 14,000 local newspapers, TV, and radio stations across all 50 states.
  arXiv  Detail & Related papers  (2024-08-08T18:33:37Z)
• News Deja Vu: Connecting Past and Present with Semantic Search [2.446672595462589]
  News Deja Vu is a novel semantic search tool for historical news articles. We show how it can be deployed to a massive scale corpus of historical, open-source news articles.
  arXiv  Detail & Related papers  (2024-06-21T18:50:57Z)
• OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text [112.60163342249682]
  We introduce OmniCorpus, a 10 billion-scale image-text interleaved dataset. Our dataset has 15 times larger scales while maintaining good data quality. We hope this could provide a solid data foundation for future multimodal model research.
  arXiv  Detail & Related papers  (2024-06-12T17:01:04Z)
• A New Korean Text Classification Benchmark for Recognizing the Political Intents in Online Newspapers [6.633601941627045]
  We present a novel Korean text classification dataset that contains various articles. Our dataset contains 12,000 news articles that may contain political intentions, from the politics section of six of the most representative newspaper organizations in South Korea. To the best of our knowledge, our paper is the most large-scale Korean news dataset that contains long text and addresses multi-task classification problems.
  arXiv  Detail & Related papers  (2023-11-03T04:59:55Z)
• American Stories: A Large-Scale Structured Text Dataset of Historical U.S. Newspapers [7.161822501147275]
  This study develops a novel, deep learning pipeline for extracting full article texts from newspaper images. It applies it to the nearly 20 million scans in Library of Congress's public domain Chronicling America collection. The pipeline includes layout detection, legibility classification, custom OCR, and association of article texts spanning multiple bounding boxes.
  arXiv  Detail & Related papers  (2023-08-24T00:24:42Z)
• A Massive Scale Semantic Similarity Dataset of Historical English [3.8073142980733]
  This study utilizes a novel source, newly digitized articles from off-copyright, local U.S. newspapers, to assemble a massive-scale semantic similarity dataset spanning 70 years from 1920 to 1989. We associate articles and their headlines by exploiting document layouts and language understanding. We then use deep neural methods to detect which articles are from the same underlying source, in the presence of substantial noise and abridgement. The HEADLINES dataset is significantly larger than most existing semantic similarity datasets and covers a much longer span of time.
  arXiv  Detail & Related papers  (2023-06-30T17:16:04Z)
• Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection [83.3580786484122]
  We find that newspapers from larger schools, located in wealthier, educated, and urban ZIP codes are more likely to be classified as high quality. We argue that privileging any corpus as high quality entails a language ideology.
  arXiv  Detail & Related papers  (2022-01-25T17:20:04Z)
• Automatic Speech Recognition Datasets in Cantonese Language: A Survey and a New Dataset [85.52036362232688]
  Our dataset consists of 73.6 hours of clean read speech paired with transcripts, collected from Cantonese audiobooks from Hong Kong. It combines philosophy, politics, education, culture, lifestyle and family domains, covering a wide range of topics. We create a powerful and robust Cantonese ASR model by applying multi-dataset learning on MDCC and Common Voice zh-HK.
  arXiv  Detail & Related papers  (2022-01-07T12:09:15Z)
• Assessing the quality of sources in Wikidata across languages: a hybrid approach [64.05097584373979]
  We run a series of microtasks experiments to evaluate a large corpus of references, sampled from Wikidata triples with labels in several languages. We use a consolidated, curated version of the crowdsourced assessments to train several machine learning models to scale up the analysis to the whole of Wikidata. The findings help us ascertain the quality of references in Wikidata, and identify common challenges in defining and capturing the quality of user-generated multilingual structured data on the web.
  arXiv  Detail & Related papers  (2021-09-20T10:06:46Z)
• \textit{NewsEdits}: A Dataset of Revision Histories for News Articles (Technical Report: Data Processing) [89.77347919191774]
  textitNewsEdits is the first publicly available dataset of news article revision histories. It contains 1,278,804 articles with 4,609,430 versions from over 22 English- and French-language newspaper sources.
  arXiv  Detail & Related papers  (2021-04-19T21:15:30Z)
• The Newspaper Navigator Dataset: Extracting And Analyzing Visual Content from 16 Million Historic Newspaper Pages in Chronicling America [10.446473806802578]
  We introduce a visual content recognition model trained on bounding box annotations of photographs, illustrations, maps, comics, and editorial cartoons. We describe our pipeline that utilizes this deep learning model to extract 7 classes of visual content. We report the results of running the pipeline on 16.3 million pages from the Chronicling America corpus.
  arXiv  Detail & Related papers  (2020-05-04T15:51:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.