Natural Language Inference with Mixed Effects

• URL: http://arxiv.org/abs/2010.10501v1
• Date: Tue, 20 Oct 2020 17:54:16 GMT
• Title: Natural Language Inference with Mixed Effects
• Authors: William Gantt, Benjamin Kane, Aaron Steven White
• Abstract summary: We propose a generic method that allows one to skip the aggregation step and train on the raw annotations directly without subjecting the model to unwanted noise. We demonstrate that this method, which generalizes the notion of a textitmixed effects model by incorporating textitannotator random effects into any existing neural model, improves performance over models that do not incorporate such effects.
• Score: 4.560556461930812
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: There is growing evidence that the prevalence of disagreement in the raw annotations used to construct natural language inference datasets makes the common practice of aggregating those annotations to a single label problematic. We propose a generic method that allows one to skip the aggregation step and train on the raw annotations directly without subjecting the model to unwanted noise that can arise from annotator response biases. We demonstrate that this method, which generalizes the notion of a \textit{mixed effects model} by incorporating \textit{annotator random effects} into any existing neural model, improves performance over models that do not incorporate such effects.

Related papers

• Testing Hypotheses of Covariate Effects on Topics of Discourse [0.0]
  We introduce an approach to topic modelling that remains tractable in the face of large text corpora.<n>This is achieved by de-emphasizing the role of parameter estimation in an underlying probabilistic model.<n>We argue that the simple, non-parametric approach advocated here is faster, more interpretable, and enjoys better inferential justification than said generative models.
  arXiv  Detail & Related papers  (2025-06-05T20:28:49Z)
• Counterfactual Generation from Language Models [64.55296662926919]
  We show that counterfactual reasoning is conceptually distinct from interventions. We propose a framework for generating true string counterfactuals. Our experiments demonstrate that the approach produces meaningful counterfactuals.
  arXiv  Detail & Related papers  (2024-11-11T17:57:30Z)
• Relation-based Counterfactual Data Augmentation and Contrastive Learning for Robustifying Natural Language Inference Models [0.0]
  We propose a method in which we use token-based and sentence-based augmentation methods to generate counterfactual sentence pairs. We show that the proposed method can improve the performance and robustness of the NLI model.
  arXiv  Detail & Related papers  (2024-10-28T03:43:25Z)
• Attacks against Abstractive Text Summarization Models through Lead Bias and Influence Functions [1.7863534204867277]
  Large Language Models are vulnerable to adversarial perturbations and data poisoning attacks. In this work, we unveil a novel approach by exploiting the inherent lead bias in summarization models. We also introduce an innovative application of influence functions, to execute data poisoning, which compromises the model's integrity.
  arXiv  Detail & Related papers  (2024-10-26T00:35:15Z)
• Contextual Biasing to Improve Domain-specific Custom Vocabulary Audio Transcription without Explicit Fine-Tuning of Whisper Model [0.0]
  OpenAI's Whisper Automated Speech Recognition model excels in generalizing across diverse datasets and domains. We propose a method to enhance transcription accuracy without explicit fine-tuning or altering model parameters.
  arXiv  Detail & Related papers  (2024-10-24T01:58:11Z)
• AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation [57.8363998797433]
  We propose AMRFact, a framework that generates perturbed summaries using Abstract Meaning Representations (AMRs) Our approach parses factually consistent summaries into AMR graphs and injects controlled factual inconsistencies to create negative examples, allowing for coherent factually inconsistent summaries to be generated with high error-type coverage.
  arXiv  Detail & Related papers  (2023-11-16T02:56:29Z)
• Assessing the overall and partial causal well-specification of nonlinear additive noise models [4.13592995550836]
  We aim to identify predictor variables for which we can infer the causal effect even in cases of such misspecifications. We propose an algorithm for finite sample data, discuss its properties, and illustrate its performance on simulated and real data.
  arXiv  Detail & Related papers  (2023-10-25T09:44:16Z)
• NaturalAdversaries: Can Naturalistic Adversaries Be as Effective as Artificial Adversaries? [61.58261351116679]
  We introduce a two-stage adversarial example generation framework (NaturalAdversaries) for natural language understanding tasks. It is adaptable to both black-box and white-box adversarial attacks based on the level of access to the model parameters. Our results indicate these adversaries generalize across domains, and offer insights for future research on improving robustness of neural text classification models.
  arXiv  Detail & Related papers  (2022-11-08T16:37:34Z)
• Rethinking Generalization: The Impact of Annotation Style on Medical Image Segmentation [9.056814157662965]
  We show that modeling annotation biases, rather than ignoring them, poses a promising way of accounting for differences in annotation style across datasets. Next, we present an image-conditioning approach to model annotation styles that correlate with specific image features, potentially enabling detection biases to be more easily identified.
  arXiv  Detail & Related papers  (2022-10-31T15:28:49Z)
• Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task [70.29624135819884]
  We study the extent to which BERT is able to perform lexically-independent subject-verb number agreement (NA) on targeted syntactic templates. Our results on nonce sentences suggest that the model generalizes well for simple templates, but fails to perform lexically-independent syntactic generalization when as little as one attractor is present.
  arXiv  Detail & Related papers  (2022-04-14T11:33:15Z)
• Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning [57.4036085386653]
  We show that prompt-based models for sentence pair classification tasks still suffer from a common pitfall of adopting inferences based on lexical overlap. We then show that adding a regularization that preserves pretraining weights is effective in mitigating this destructive tendency of few-shot finetuning.
  arXiv  Detail & Related papers  (2021-09-09T10:10:29Z)
• Accounting for Unobserved Confounding in Domain Generalization [107.0464488046289]
  This paper investigates the problem of learning robust, generalizable prediction models from a combination of datasets. Part of the challenge of learning robust models lies in the influence of unobserved confounders. We demonstrate the empirical performance of our approach on healthcare data from different modalities.
  arXiv  Detail & Related papers  (2020-07-21T08:18:06Z)
• Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing [83.78668073898001]
  We introduce a family of entropy regularizers, which includes label smoothing as a special case. We find that variance in model performance can be explained largely by the resulting entropy of the model. We advise the use of other entropy regularization methods in its place.
  arXiv  Detail & Related papers  (2020-05-02T12:46:28Z)
• Learning to Manipulate Individual Objects in an Image [71.55005356240761]
  We describe a method to train a generative model with latent factors that are independent and localized. This means that perturbing the latent variables affects only local regions of the synthesized image, corresponding to objects. Unlike other unsupervised generative models, ours enables object-centric manipulation, without requiring object-level annotations.
  arXiv  Detail & Related papers  (2020-04-11T21:50:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.