Metadata Might Make Language Models Better

• URL: http://arxiv.org/abs/2211.10086v1
• Date: Fri, 18 Nov 2022 08:29:00 GMT
• Title: Metadata Might Make Language Models Better
• Authors: Kaspar Beelen and Daniel van Strien
• Abstract summary: Using 19th-century newspapers as a case study, we compare different strategies for inserting temporal, political and geographical information into a Masked Language Model. We find that showing relevant metadata to a language model has a beneficial impact and may even produce more robust and fairer models.
• Score: 1.7100280218774935
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper discusses the benefits of including metadata when training language models on historical collections. Using 19th-century newspapers as a case study, we extend the time-masking approach proposed by Rosin et al., 2022 and compare different strategies for inserting temporal, political and geographical information into a Masked Language Model. After fine-tuning several DistilBERT on enhanced input data, we provide a systematic evaluation of these models on a set of evaluation tasks: pseudo-perplexity, metadata mask-filling and supervised classification. We find that showing relevant metadata to a language model has a beneficial impact and may even produce more robust and fairer models.

Related papers

• Improving Pinterest Search Relevance Using Large Language Models [15.24121687428178]
  We integrate Large Language Models (LLMs) into our search relevance model. Our approach uses search queries alongside content representations that include captions extracted from a generative visual language model. We distill from the LLM-based model into real-time servable model architectures and features.
  arXiv  Detail & Related papers  (2024-10-22T16:29:33Z)
• Multilingual Models for Check-Worthy Social Media Posts Detection [0.552480439325792]
  The study includes a comprehensive analysis of different models, with a special focus on multilingual models. The novelty of this work lies in the development of multi-label multilingual classification models that can simultaneously detect harmful posts and posts that contain verifiable factual claims in an efficient way.
  arXiv  Detail & Related papers  (2024-08-13T08:55:28Z)
• Masked Image Modeling: A Survey [73.21154550957898]
  Masked image modeling emerged as a powerful self-supervised learning technique in computer vision. We construct a taxonomy and review the most prominent papers in recent years. We aggregate the performance results of various masked image modeling methods on the most popular datasets.
  arXiv  Detail & Related papers  (2024-08-13T07:27:02Z)
• Learning Phonotactics from Linguistic Informants [54.086544221761486]
  Our model iteratively selects or synthesizes a data-point according to one of a range of information-theoretic policies. We find that the information-theoretic policies that our model uses to select items to query the informant achieve sample efficiency comparable to, or greater than, fully supervised approaches.
  arXiv  Detail & Related papers  (2024-05-08T00:18:56Z)
• PRobELM: Plausibility Ranking Evaluation for Language Models [12.057770969325453]
  PRobELM is a benchmark designed to assess language models' ability to discern more plausible scenarios through their parametric knowledge. Our benchmark is constructed from a dataset curated from Wikidata edit histories, tailored to align the temporal bounds of the training data for the evaluated models.
  arXiv  Detail & Related papers  (2024-04-04T21:57:11Z)
• Revisiting Topic-Guided Language Models [20.21486464604549]
  We study four topic-guided language models and two baselines, evaluating the held-out predictive performance of each model on four corpora. We find that none of these methods outperform a standard LSTM language model baseline, and most fail to learn good topics.
  arXiv  Detail & Related papers  (2023-12-04T20:33:24Z)
• Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks [65.21536453075275]
  We focus on the summarization task and investigate the membership inference (MI) attack. We exploit text similarity and the model's resistance to document modifications as potential MI signals. We discuss several safeguards for training summarization models to protect against MI attacks and discuss the inherent trade-off between privacy and utility.
  arXiv  Detail & Related papers  (2023-10-20T05:44:39Z)
• Pre-trained Language Model with Prompts for Temporal Knowledge Graph Completion [30.50032335014021]
  We propose a novel TKGC model, namely Pre-trained Language Model with Prompts for TKGC (PPT) We convert a series of sampled quadruples into pre-trained language model inputs and convert intervals between timestamps into different prompts to make coherent sentences with implicit semantic information. Our model can effectively incorporate information from temporal knowledge graphs into the language models.
  arXiv  Detail & Related papers  (2023-05-13T12:53:11Z)
• A Generative Language Model for Few-shot Aspect-Based Sentiment Analysis [90.24921443175514]
  We focus on aspect-based sentiment analysis, which involves extracting aspect term, category, and predicting their corresponding polarities. We propose to reformulate the extraction and prediction tasks into the sequence generation task, using a generative language model with unidirectional attention. Our approach outperforms the previous state-of-the-art (based on BERT) on average performance by a large margins in few-shot and full-shot settings.
  arXiv  Detail & Related papers  (2022-04-11T18:31:53Z)
• Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model [58.27176041092891]
  Recent research indicates that pretraining cross-lingual language models on large-scale unlabeled texts yields significant performance improvements. We propose a novel unsupervised feature decomposition method that can automatically extract domain-specific features from the entangled pretrained cross-lingual representations. Our proposed model leverages mutual information estimation to decompose the representations computed by a cross-lingual model into domain-invariant and domain-specific parts.
  arXiv  Detail & Related papers  (2020-11-23T16:00:42Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.