Hallucination, Monofacts, and Miscalibration: An Empirical Investigation

• URL: http://arxiv.org/abs/2502.08666v1
• Date: Tue, 11 Feb 2025 18:46:00 GMT
• Title: Hallucination, Monofacts, and Miscalibration: An Empirical Investigation
• Authors: Muqing Miao, Michael Kearns,
• Abstract summary: We show how different underlying data distributions affect the monofact rate and a model's tendency to hallucinate.<n>These findings suggest that both the distribution of fact frequencies in training data and the calibration-hallucination trade-off are inherent to probabilistic language generation.
• Score: 2.6162433502464757
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent theoretical work by [Kalai and Vempala 2024] proves that a particular notion of hallucination rate in LLMs must be lower bounded by the training data monofact rate (related to the classical Good-Turing missing mass estimator) minus model miscalibration. Through systematic experiments with n-gram models and in-context learning with LLMs, we empirically investigate and validate this theory by examining how different underlying data distributions affect the monofact rate and a model's tendency to hallucinate. We then vary model miscalibration through controlled upweighting of training samples while holding monofact rates constant, allowing us to isolate miscalibration's reduction effect on hallucination. These findings suggest that both the distribution of fact frequencies in training data and the calibration-hallucination trade-off are inherent to probabilistic language generation. Our results also suggest that current practices of aggressive deduplication in training data may need to be reconsidered, as selective duplication could serve as a principled mechanism for reducing hallucination.

Related papers

• Causal Lifting of Neural Representations: Zero-Shot Generalization for Causal Inferences [56.23412698865433]
  We focus on causal inferences on a target experiment with unlabeled factual outcomes, retrieved by a predictive model fine-tuned on a labeled similar experiment.<n>First, we show that factual outcome estimation via Empirical Risk Minimization (ERM) may fail to yield valid causal inferences on the target population.<n>We propose Deconfounded Empirical Risk Minimization (DERM), a new simple learning procedure minimizing the risk over a fictitious target population.
  arXiv  Detail & Related papers  (2025-02-10T10:52:17Z)
• Hallucination Detox: Sensitivity Dropout (SenD) for Large Language Model Training [7.726825072908519]
  This research investigates the relationship between the training process and the emergence of hallucinations.<n>We introduce Sensitivity Dropout (SenD), a novel training protocol designed to mitigate hallucinations by reducing variance during training.<n>In addition, we develop an unsupervised hallucination detection metric, Efficient EigenScore (EES), which approximates the traditional EigenScore at 2x speed.
  arXiv  Detail & Related papers  (2024-10-20T18:18:23Z)
• Low-rank finetuning for LLMs: A fairness perspective [54.13240282850982]
  Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models. This paper investigates the effectiveness of these methods in capturing the shift of fine-tuning datasets from the initial pre-trained data distribution. We show that low-rank fine-tuning inadvertently preserves undesirable biases and toxic behaviors.
  arXiv  Detail & Related papers  (2024-05-28T20:43:53Z)
• Prescribing the Right Remedy: Mitigating Hallucinations in Large Vision-Language Models via Targeted Instruction Tuning [15.156359255401812]
  We propose a targeted instruction data generation framework named DFTG that tailored to the hallucination specificity of different models. The experimental results on hallucination benchmarks demonstrate that the targeted instruction data generated by our method are more effective in mitigating hallucinations compared to previous datasets.
  arXiv  Detail & Related papers  (2024-04-16T07:14:32Z)
• Unfamiliar Finetuning Examples Control How Language Models Hallucinate [75.03210107477157]
  Large language models are known to hallucinate when faced with unfamiliar queries. We find that unfamiliar examples in the models' finetuning data are crucial in shaping these errors. Our work further investigates RL finetuning strategies for improving the factuality of long-form model generations.
  arXiv  Detail & Related papers  (2024-03-08T18:28:13Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• Data Attribution for Diffusion Models: Timestep-induced Bias in Influence Estimation [53.27596811146316]
  Diffusion models operate over a sequence of timesteps instead of instantaneous input-output relationships in previous contexts. We present Diffusion-TracIn that incorporates this temporal dynamics and observe that samples' loss gradient norms are highly dependent on timestep. We introduce Diffusion-ReTrac as a re-normalized adaptation that enables the retrieval of training samples more targeted to the test sample of interest.
  arXiv  Detail & Related papers  (2024-01-17T07:58:18Z)
• Delving into Semantic Scale Imbalance [45.30062061215943]
  We define and quantify the semantic scale of classes, which is used to measure the feature diversity of classes. We propose semantic-scale-balanced learning, including a general loss improvement scheme and a dynamic re-weighting training framework. Comprehensive experiments show that dynamic semantic-scale-balanced learning consistently enables the model to perform superiorly on large-scale long-tailed and non-long-tailed natural and medical datasets.
  arXiv  Detail & Related papers  (2022-12-30T09:40:09Z)
• Efficient Causal Inference from Combined Observational and Interventional Data through Causal Reductions [68.6505592770171]
  Unobserved confounding is one of the main challenges when estimating causal effects. We propose a novel causal reduction method that replaces an arbitrary number of possibly high-dimensional latent confounders. We propose a learning algorithm to estimate the parameterized reduced model jointly from observational and interventional data.
  arXiv  Detail & Related papers  (2021-03-08T14:29:07Z)
• Bayesian Sampling Bias Correction: Training with the Right Loss Function [0.0]
  We derive a family of loss functions to train models in the presence of sampling bias. Examples are when the prevalence of a pathology differs from its sampling rate in the training dataset, or when a machine learning practioner rebalances their training dataset.
  arXiv  Detail & Related papers  (2020-06-24T15:10:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.