News Deja Vu: Connecting Past and Present with Semantic Search

• URL: http://arxiv.org/abs/2406.15593v1
• Date: Fri, 21 Jun 2024 18:50:57 GMT
• Title: News Deja Vu: Connecting Past and Present with Semantic Search
• Authors: Brevin Franklin, Emily Silcock, Abhishek Arora, Tom Bryan, Melissa Dell,
• Abstract summary: News Deja Vu is a novel semantic search tool for historical news articles. We show how it can be deployed to a massive scale corpus of historical, open-source news articles.
• Score: 2.446672595462589
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Social scientists and the general public often analyze contemporary events by drawing parallels with the past, a process complicated by the vast, noisy, and unstructured nature of historical texts. For example, hundreds of millions of page scans from historical newspapers have been noisily transcribed. Traditional sparse methods for searching for relevant material in these vast corpora, e.g., with keywords, can be brittle given complex vocabularies and OCR noise. This study introduces News Deja Vu, a novel semantic search tool that leverages transformer large language models and a bi-encoder approach to identify historical news articles that are most similar to modern news queries. News Deja Vu first recognizes and masks entities, in order to focus on broader parallels rather than the specific named entities being discussed. Then, a contrastively trained, lightweight bi-encoder retrieves historical articles that are most similar semantically to a modern query, illustrating how phenomena that might seem unique to the present have varied historical precedents. Aimed at social scientists, the user-friendly News Deja Vu package is designed to be accessible for those who lack extensive familiarity with deep learning. It works with large text datasets, and we show how it can be deployed to a massive scale corpus of historical, open-source news articles. While human expertise remains important for drawing deeper insights, News Deja Vu provides a powerful tool for exploring parallels in how people have perceived past and present.

Related papers

• Contrastive Entity Coreference and Disambiguation for Historical Texts [2.446672595462589]
  Existing entity disambiguation methods often fall short in accuracy for historical documents, which are replete with individuals not remembered in contemporary knowledgebases. This study makes three key contributions to improve cross-document coreference resolution and disambiguation in historical texts.
  arXiv  Detail & Related papers  (2024-06-21T18:22:14Z)
• Newswire: A Large-Scale Structured Database of a Century of Historical News [3.562368079040469]
  Historians argue that newswires played a pivotal role in creating a national identity and shared understanding of the world. We reconstruct such an archive by applying a customized deep learning pipeline to hundreds of terabytes of raw image scans from thousands of local newspapers. The resulting dataset contains 2.7 million unique public domain U.S. newswire articles, written between 1878 and 1977.
  arXiv  Detail & Related papers  (2024-06-13T16:20:05Z)
• SCStory: Self-supervised and Continual Online Story Discovery [53.72745249384159]
  SCStory helps people digest rapidly published news article streams in real-time without human annotations. SCStory employs self-supervised and continual learning with a novel idea of story-indicative adaptive modeling of news article streams.
  arXiv  Detail & Related papers  (2023-11-27T04:50:01Z)
• American Stories: A Large-Scale Structured Text Dataset of Historical U.S. Newspapers [7.161822501147275]
  This study develops a novel, deep learning pipeline for extracting full article texts from newspaper images. It applies it to the nearly 20 million scans in Library of Congress's public domain Chronicling America collection. The pipeline includes layout detection, legibility classification, custom OCR, and association of article texts spanning multiple bounding boxes.
  arXiv  Detail & Related papers  (2023-08-24T00:24:42Z)
• A Massive Scale Semantic Similarity Dataset of Historical English [3.8073142980733]
  This study utilizes a novel source, newly digitized articles from off-copyright, local U.S. newspapers, to assemble a massive-scale semantic similarity dataset spanning 70 years from 1920 to 1989. We associate articles and their headlines by exploiting document layouts and language understanding. We then use deep neural methods to detect which articles are from the same underlying source, in the presence of substantial noise and abridgement. The HEADLINES dataset is significantly larger than most existing semantic similarity datasets and covers a much longer span of time.
  arXiv  Detail & Related papers  (2023-06-30T17:16:04Z)
• Towards Open Vocabulary Learning: A Survey [146.90188069113213]
  Deep neural networks have made impressive advancements in various core tasks like segmentation, tracking, and detection. Recently, open vocabulary settings were proposed due to the rapid progress of vision language pre-training. This paper provides a thorough review of open vocabulary learning, summarizing and analyzing recent developments in the field.
  arXiv  Detail & Related papers  (2023-06-28T02:33:06Z)
• History-Aware Hierarchical Transformer for Multi-session Open-domain Dialogue System [59.78425104243993]
  We propose History-Aware Hierarchical Transformer (HAHT) for multi-session open-domain dialogue. HAHT maintains a long-term memory of history conversations and utilizes history information to understand current conversation context. Experimental results on a large-scale Multi-Session Conversation dataset suggest that the proposed HAHT model consistently outperforms baseline models.
  arXiv  Detail & Related papers  (2023-02-02T06:54:33Z)
• Focus! Relevant and Sufficient Context Selection for News Image Captioning [69.36678144800936]
  News Image Captioning requires describing an image by leveraging additional context from a news article. We propose to use the pre-trained vision and language retrieval model CLIP to localize the visually grounded entities in the news article. Our experiments demonstrate that by simply selecting a better context from the article, we can significantly improve the performance of existing models.
  arXiv  Detail & Related papers  (2022-12-01T20:00:27Z)
• Assessing the quality of sources in Wikidata across languages: a hybrid approach [64.05097584373979]
  We run a series of microtasks experiments to evaluate a large corpus of references, sampled from Wikidata triples with labels in several languages. We use a consolidated, curated version of the crowdsourced assessments to train several machine learning models to scale up the analysis to the whole of Wikidata. The findings help us ascertain the quality of references in Wikidata, and identify common challenges in defining and capturing the quality of user-generated multilingual structured data on the web.
  arXiv  Detail & Related papers  (2021-09-20T10:06:46Z)
• Deep Learning for Text Style Transfer: A Survey [71.8870854396927]
  Text style transfer is an important task in natural language generation, which aims to control certain attributes in the generated text. We present a systematic survey of the research on neural text style transfer, spanning over 100 representative articles since the first neural text style transfer work in 2017. We discuss the task formulation, existing datasets and subtasks, evaluation, as well as the rich methodologies in the presence of parallel and non-parallel data.
  arXiv  Detail & Related papers  (2020-11-01T04:04:43Z)
• Combining Visual and Textual Features for Semantic Segmentation of Historical Newspapers [2.5899040911480187]
  We introduce a multimodal approach for the semantic segmentation of historical newspapers. Based on experiments on diachronic Swiss and Luxembourgish newspapers, we investigate the predictive power of visual and textual features. Results show consistent improvement of multimodal models in comparison to a strong visual baseline.
  arXiv  Detail & Related papers  (2020-02-14T17:56:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.