Product-of-Experts Training Reduces Dataset Artifacts in Natural Language Inference

• URL: http://arxiv.org/abs/2604.19069v1
• Date: Tue, 21 Apr 2026 04:23:20 GMT
• Title: Product-of-Experts Training Reduces Dataset Artifacts in Natural Language Inference
• Abstract summary: We propose Product-of-Experts (PoE) training, which downweights examples where biased models are overconfident.<n>PoE nearly preserves accuracy (89.10% vs. 89.30%) while cutting bias by 4.71% (bias agreement 49.85% to 45%)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural NLI models overfit dataset artifacts instead of truly reasoning. A hypothesis-only model gets 57.7% in SNLI, showing strong spurious correlations, and 38.6% of the baseline errors are the result of these artifacts. We propose Product-of-Experts (PoE) training, which downweights examples where biased models are overconfident. PoE nearly preserves accuracy (89.10% vs. 89.30%) while cutting bias reliance by 4.71% (bias agreement 49.85% to 45%). An ablation finds lambda = 1.5 that best balances debiasing and accuracy. Behavioral tests still reveal issues with negation and numerical reasoning.

Related papers

• Theory-Guided Deception Detection: A RAG-Based Artificial Intelligence Exploration [0.14323566945483496]
  Work developed seven Retrieval-Augmented Generation (RAG) models based on leading deception theories.<n>Analysed how deception judgments were made relative to baseline models.
  arXiv  Detail & Related papers  (2026-08-09T19:39:19Z)
• Bayesian Elicitation with LLMs: Model Size Helps, Extra "Reasoning" Doesn't Always [0.0]
  Large language models (LLMs) have been proposed as alternatives to human experts for estimating unknown quantities with associated uncertainty.<n>We test this by asking eleven LLMs to estimate population statistics, such as health prevalence rates, personality trait distributions, and labor market figures.<n>We vary each model's reasoning effort to test whether more "thinking" improves results.
  arXiv  Detail & Related papers  (2026-04-02T11:00:13Z)
• Causal Understanding by LLMs: The Role of Uncertainty [43.87879175532034]
  Recent papers show LLMs achieve near-random accuracy in causal relation classification.<n>We investigate whether pretraining exposure to causal examples improves causal understanding.
  arXiv  Detail & Related papers  (2025-09-24T13:06:35Z)
• Flattery, Fluff, and Fog: Diagnosing and Mitigating Idiosyncratic Biases in Preference Models [12.445845925904466]
  Language models serve as proxies for human preference judgements in alignment and evaluation.<n>They exhibit systematic miscalibration, prioritizing superficial patterns over substantive qualities.<n>This bias manifests as overreliance on features like length, structure, and style, leading to issues like reward hacking and unreliable evaluations.
  arXiv  Detail & Related papers  (2025-06-05T17:59:32Z)
• Multi-head attention debiasing and contrastive learning for mitigating Dataset Artifacts in Natural Language Inference [0.0]
  We develop a novel structural debiasing approach for Natural Language Inference models.<n>Our approach reduces the error rate from 14.19% to 10.42% while maintaining high performance on unbiased examples.
  arXiv  Detail & Related papers  (2024-12-16T17:12:21Z)
• Stubborn Lexical Bias in Data and Models [50.79738900885665]
  We use a new statistical method to examine whether spurious patterns in data appear in models trained on the data. We apply an optimization approach to *reweight* the training data, reducing thousands of spurious correlations. Surprisingly, though this method can successfully reduce lexical biases in the training data, we still find strong evidence of corresponding bias in the trained models.
  arXiv  Detail & Related papers  (2023-06-03T20:12:27Z)
• Impact of Pretraining Term Frequencies on Few-Shot Reasoning [51.990349528930125]
  We investigate how well pretrained language models reason with terms that are less frequent in the pretraining data. We measure the strength of this correlation for a number of GPT-based language models on various numerical deduction tasks. Although LMs exhibit strong performance at few-shot numerical reasoning tasks, our results raise the question of how much models actually generalize beyond pretraining data.
  arXiv  Detail & Related papers  (2022-02-15T05:43:54Z)
• Learning from others' mistakes: Avoiding dataset biases without modeling them [111.17078939377313]
  State-of-the-art natural language processing (NLP) models often learn to model dataset biases and surface form correlations instead of features that target the intended task. Previous work has demonstrated effective methods to circumvent these issues when knowledge of the bias is available. We show a method for training models that learn to ignore these problematic correlations.
  arXiv  Detail & Related papers  (2020-12-02T16:10:54Z)
• What Can We Learn from Collective Human Opinions on Natural Language Inference Data? [88.90490998032429]
  ChaosNLI is a dataset with a total of 464,500 annotations to study Collective HumAn OpinionS. This dataset is created by collecting 100 annotations per example for 3,113 examples in SNLI and MNLI and 1,532 examples in Abductive-NLI.
  arXiv  Detail & Related papers  (2020-10-07T17:26:06Z)
• Towards Robustifying NLI Models Against Lexical Dataset Biases [94.79704960296108]
  This paper explores both data-level and model-level debiasing methods to robustify models against lexical dataset biases. First, we debias the dataset through data augmentation and enhancement, but show that the model bias cannot be fully removed via this method. The second approach employs a bag-of-words sub-model to capture the features that are likely to exploit the bias and prevents the original model from learning these biased features.
  arXiv  Detail & Related papers  (2020-05-10T17:56:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.