Annotation Tool and Dataset for Fact-Checking Podcasts

• URL: http://arxiv.org/abs/2502.01402v1
• Date: Mon, 03 Feb 2025 14:34:17 GMT
• Title: Annotation Tool and Dataset for Fact-Checking Podcasts
• Authors: Vinay Setty, Adam James Becker,
• Abstract summary: podcasts are a popular medium on the web, featuring diverse and multilingual content that often includes unverified claims. Our tool offers a novel approach to tackle these challenges by enabling real-time annotation of contextual during playback. This unique capability allows users to listen to the podcast and annotate key elements, such as check-worthy claims, claim spans, and contextual errors, simultaneously.
• Score: 1.6804613362826175
• License:
• Abstract: Podcasts are a popular medium on the web, featuring diverse and multilingual content that often includes unverified claims. Fact-checking podcasts is a challenging task, requiring transcription, annotation, and claim verification, all while preserving the contextual details of spoken content. Our tool offers a novel approach to tackle these challenges by enabling real-time annotation of podcasts during playback. This unique capability allows users to listen to the podcast and annotate key elements, such as check-worthy claims, claim spans, and contextual errors, simultaneously. By integrating advanced transcription models like OpenAI's Whisper and leveraging crowdsourced annotations, we create high-quality datasets to fine-tune multilingual transformer models such as XLM-RoBERTa for tasks like claim detection and stance classification. Furthermore, we release the annotated podcast transcripts and sample annotations with preliminary experiments.

Related papers

• Can Language Models Learn to Listen? [96.01685069483025]
  We present a framework for generating appropriate facial responses from a listener in dyadic social interactions based on the speaker's words. Our approach autoregressively predicts a response of a listener: a sequence of listener facial gestures, quantized using a VQ-VAE. We show that our generated listener motion is fluent and reflective of language semantics through quantitative metrics and a qualitative user study.
  arXiv  Detail & Related papers  (2023-08-21T17:59:02Z)
• WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research [82.42802570171096]
  We introduce WavCaps, the first large-scale weakly-labelled audio captioning dataset, comprising approximately 400k audio clips with paired captions. Online-harvested raw descriptions are highly noisy and unsuitable for direct use in tasks such as automated audio captioning. We propose a three-stage processing pipeline for filtering noisy data and generating high-quality captions, where ChatGPT, a large language model, is leveraged to filter and transform raw descriptions automatically.
  arXiv  Detail & Related papers  (2023-03-30T14:07:47Z)
• WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Models [57.557319372969495]
  Large-scale auto-regressive language models pretrained on massive text have demonstrated their impressive ability to perform new natural language tasks. Recent studies further show that such a few-shot learning ability can be extended to the text-image setting by training an encoder to encode the images into embeddings. We propose a novel speech understanding framework, WavPrompt, where we finetune a wav2vec model to generate a sequence of audio embeddings understood by the language model.
  arXiv  Detail & Related papers  (2022-03-29T19:08:55Z)
• Towards Abstractive Grounded Summarization of Podcast Transcripts [33.268079036601634]
  Summarization of podcast transcripts is of practical benefit to both content providers and consumers. It helps consumers to quickly decide whether they will listen to the podcasts and reduces the load of content providers to write summaries. However, podcast summarization faces significant challenges including factual inconsistencies with respect to the inputs.
  arXiv  Detail & Related papers  (2022-03-22T02:44:39Z)
• Identifying Introductions in Podcast Episodes from Automatically Generated Transcripts [0.0]
  We build a novel dataset of complete transcriptions of over 400 podcast episodes. These introductions contain information about the episodes' topics, hosts, and guests. We train three Transformer models based on the pre-trained BERT and different augmentation strategies.
  arXiv  Detail & Related papers  (2021-10-14T00:34:51Z)
• Topic Model Robustness to Automatic Speech Recognition Errors in Podcast Transcripts [4.526933031343007]
  In this work, we explore the robustness of a Latent Dirichlet Allocation topic model when applied to transcripts created by an automatic speech recognition engine. First, we observe a baseline of cosine similarity scores between topic embeddings from automatic transcriptions and the descriptions of the podcasts written by the podcast creators. We then observe how the cosine similarities decrease as transcription noise increases and conclude that even when automatic speech recognition transcripts are erroneous, it is still possible to obtain high-quality topic embeddings from the transcriptions.
  arXiv  Detail & Related papers  (2021-09-25T07:59:31Z)
• Annotation Curricula to Implicitly Train Non-Expert Annotators [56.67768938052715]
  voluntary studies often require annotators to familiarize themselves with the task, its annotation scheme, and the data domain. This can be overwhelming in the beginning, mentally taxing, and induce errors into the resulting annotations. We propose annotation curricula, a novel approach to implicitly train annotators.
  arXiv  Detail & Related papers  (2021-06-04T09:48:28Z)
• Detecting Extraneous Content in Podcasts [6.335863593761816]
  We present a model that leverage both textual and listening patterns to detect extraneous content in podcast descriptions and audio transcripts. We show that our models can substantively improve ROUGE scores and reduce the extraneous content generated in the summaries.
  arXiv  Detail & Related papers  (2021-03-03T18:30:50Z)
• Watch and Learn: Mapping Language and Noisy Real-world Videos with Self-supervision [54.73758942064708]
  We teach machines to understand visuals and natural language by learning the mapping between sentences and noisy video snippets without explicit annotations. For training and evaluation, we contribute a new dataset ApartmenTour' that contains a large number of online videos and subtitles.
  arXiv  Detail & Related papers  (2020-11-19T03:43:56Z)
• PodSumm -- Podcast Audio Summarization [0.0]
  We propose a method to automatically construct a podcast summary via guidance from the text-domain. Motivated by a lack of datasets for this task, we curate an internal dataset, find an effective scheme for data augmentation, and design a protocol to gather summaries from annotators. Our method achieves ROUGE-F(1/2/L) scores of 0.63/0.53/0.63 on our dataset.
  arXiv  Detail & Related papers  (2020-09-22T04:49:33Z)
• Abstractive Summarization of Spoken and Written Instructions with BERT [66.14755043607776]
  We present the first application of the BERTSum model to conversational language. We generate abstractive summaries of narrated instructional videos across a wide variety of topics. We envision this integrated as a feature in intelligent virtual assistants, enabling them to summarize both written and spoken instructional content upon request.
  arXiv  Detail & Related papers  (2020-08-21T20:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.