Sorting through the noise: Testing robustness of information processing in pre-trained language models

• URL: http://arxiv.org/abs/2109.12393v1
• Date: Sat, 25 Sep 2021 16:02:23 GMT
• Title: Sorting through the noise: Testing robustness of information processing in pre-trained language models
• Authors: Lalchand Pandia and Allyson Ettinger
• Abstract summary: This paper examines robustness of models' ability to deploy relevant context information in the face of distracting content. We find that although models appear in simple contexts to make predictions based on understanding and applying relevant facts from prior context, the presence of distracting but irrelevant content has clear impact in confusing model predictions.
• Score: 5.371816551086117
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained LMs have shown impressive performance on downstream NLP tasks, but we have yet to establish a clear understanding of their sophistication when it comes to processing, retaining, and applying information presented in their input. In this paper we tackle a component of this question by examining robustness of models' ability to deploy relevant context information in the face of distracting content. We present models with cloze tasks requiring use of critical context information, and introduce distracting content to test how robustly the models retain and use that critical information for prediction. We also systematically manipulate the nature of these distractors, to shed light on dynamics of models' use of contextual cues. We find that although models appear in simple contexts to make predictions based on understanding and applying relevant facts from prior context, the presence of distracting but irrelevant content has clear impact in confusing model predictions. In particular, models appear particularly susceptible to factors of semantic similarity and word position. The findings are consistent with the conclusion that LM predictions are driven in large part by superficial contextual cues, rather than by robust representations of context meaning.

Related papers

• Context is Key: A Benchmark for Forecasting with Essential Textual Information [87.3175915185287]
  "Context is Key" (CiK) is a time series forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context. We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters. Our experiments highlight the importance of incorporating contextual information, demonstrate surprising performance when using LLM-based forecasting models, and also reveal some of their critical shortcomings.
  arXiv  Detail & Related papers  (2024-10-24T17:56:08Z)
• Aggregation Artifacts in Subjective Tasks Collapse Large Language Models' Posteriors [74.04775677110179]
  In-context Learning (ICL) has become the primary method for performing natural language tasks with Large Language Models (LLMs) In this work, we examine whether this is the result of the aggregation used in corresponding datasets, where trying to combine low-agreement, disparate annotations might lead to annotation artifacts that create detrimental noise in the prompt. Our results indicate that aggregation is a confounding factor in the modeling of subjective tasks, and advocate focusing on modeling individuals instead.
  arXiv  Detail & Related papers  (2024-10-17T17:16:00Z)
• Enhancing AI-based Generation of Software Exploits with Contextual Information [9.327315119028809]
  The study employs a dataset comprising real shellcodes to evaluate the models across various scenarios. The experiments are designed to assess the models' resilience against incomplete descriptions, their proficiency in leveraging context for enhanced accuracy, and their ability to discern irrelevant information. The models demonstrate an ability to filter out unnecessary context, maintaining high levels of accuracy in the generation of offensive security code.
  arXiv  Detail & Related papers  (2024-08-05T11:52:34Z)
• SINC: Self-Supervised In-Context Learning for Vision-Language Tasks [64.44336003123102]
  We propose a framework to enable in-context learning in large language models. A meta-model can learn on self-supervised prompts consisting of tailored demonstrations. Experiments show that SINC outperforms gradient-based methods in various vision-language tasks.
  arXiv  Detail & Related papers  (2023-07-15T08:33:08Z)
• A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis [128.0532113800092]
  We present a mechanistic interpretation of Transformer-based LMs on arithmetic questions. This provides insights into how information related to arithmetic is processed by LMs.
  arXiv  Detail & Related papers  (2023-05-24T11:43:47Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Layer-wise Analysis of a Self-supervised Speech Representation Model [26.727775920272205]
  Self-supervised learning approaches have been successful for pre-training speech representation models. Not much has been studied about the type or extent of information encoded in the pre-trained representations themselves.
  arXiv  Detail & Related papers  (2021-07-10T02:13:25Z)
• CausaLM: Causal Model Explanation Through Counterfactual Language Models [33.29636213961804]
  CausaLM is a framework for producing causal model explanations using counterfactual language representation models. We show that language representation models such as BERT can effectively learn a counterfactual representation for a given concept of interest. A byproduct of our method is a language representation model that is unaffected by the tested concept.
  arXiv  Detail & Related papers  (2020-05-27T15:06:35Z)
• Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions [55.660255727031725]
  Influence functions explain the decisions of a model by identifying influential training examples. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. We develop a new measure based on influence functions that can reveal artifacts in training data.
  arXiv  Detail & Related papers  (2020-05-14T00:45:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.