Hallucination, Monofacts, and Miscalibration: An Empirical Investigation

• URL: http://arxiv.org/abs/2502.08666v1
• Date: Tue, 11 Feb 2025 18:46:00 GMT
• Title: Hallucination, Monofacts, and Miscalibration: An Empirical Investigation
• Authors: Muqing Miao, Michael Kearns,
• Abstract summary: We show how different underlying data distributions affect the monofact rate and a model's tendency to hallucinate.<n>These findings suggest that both the distribution of fact frequencies in training data and the calibration-hallucination trade-off are inherent to probabilistic language generation.
• Score: 2.6162433502464757
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent theoretical work by [Kalai and Vempala 2024] proves that a particular notion of hallucination rate in LLMs must be lower bounded by the training data monofact rate (related to the classical Good-Turing missing mass estimator) minus model miscalibration. Through systematic experiments with n-gram models and in-context learning with LLMs, we empirically investigate and validate this theory by examining how different underlying data distributions affect the monofact rate and a model's tendency to hallucinate. We then vary model miscalibration through controlled upweighting of training samples while holding monofact rates constant, allowing us to isolate miscalibration's reduction effect on hallucination. These findings suggest that both the distribution of fact frequencies in training data and the calibration-hallucination trade-off are inherent to probabilistic language generation. Our results also suggest that current practices of aggressive deduplication in training data may need to be reconsidered, as selective duplication could serve as a principled mechanism for reducing hallucination.

Related papers

• Counterfactual Probing for Hallucination Detection and Mitigation in Large Language Models [0.0]
  We propose Counterfactual Probing, a novel approach for detecting and mitigating hallucinations in large language models.<n>Our method dynamically generates counterfactual statements that appear plausible but contain subtle factual errors, then evaluates the model's sensitivity to these perturbations.
  arXiv  Detail & Related papers  (2025-08-03T17:29:48Z)
• Shaking to Reveal: Perturbation-Based Detection of LLM Hallucinations [25.18901449626428]
  A widely adopted strategy to detect hallucination, known as self-assessment, relies on the model's own output confidence to estimate the factual accuracy of its answers.<n>We propose Sample-Specific Prompting (SSP), a new framework that improves self-assessment by analyzing perturbation sensitivity at intermediate representations.<n>SSP significantly outperforms prior methods across a range of hallucination detection benchmarks.
  arXiv  Detail & Related papers  (2025-06-03T09:44:28Z)
• Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling [67.14942827452161]
  Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations.<n>In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification.
  arXiv  Detail & Related papers  (2025-04-17T17:59:22Z)
• Causal Lifting of Neural Representations: Zero-Shot Generalization for Causal Inferences [56.23412698865433]
  We focus on causal inferences on a target experiment with unlabeled factual outcomes, retrieved by a predictive model fine-tuned on a labeled similar experiment.<n>First, we show that factual outcome estimation via Empirical Risk Minimization (ERM) may fail to yield valid causal inferences on the target population.<n>We propose Deconfounded Empirical Risk Minimization (DERM), a new simple learning procedure minimizing the risk over a fictitious target population.
  arXiv  Detail & Related papers  (2025-02-10T10:52:17Z)
• Hallucination Detox: Sensitivity Dropout (SenD) for Large Language Model Training [7.726825072908519]
  This research investigates the relationship between the training process and the emergence of hallucinations.<n>We introduce Sensitivity Dropout (SenD), a novel training protocol designed to mitigate hallucinations by reducing variance during training.<n>In addition, we develop an unsupervised hallucination detection metric, Efficient EigenScore (EES), which approximates the traditional EigenScore at 2x speed.
  arXiv  Detail & Related papers  (2024-10-20T18:18:23Z)
• Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models [65.32990889402927]
  We coin this phenomenon as knowledge overshadowing'' We show that the hallucination rate grows with both the imbalance ratio and the length of dominant condition description. We propose to utilize overshadowing conditions as a signal to catch hallucination before it is produced.
  arXiv  Detail & Related papers  (2024-07-10T20:37:42Z)
• Low-rank finetuning for LLMs: A fairness perspective [54.13240282850982]
  Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models. This paper investigates the effectiveness of these methods in capturing the shift of fine-tuning datasets from the initial pre-trained data distribution. We show that low-rank fine-tuning inadvertently preserves undesirable biases and toxic behaviors.
  arXiv  Detail & Related papers  (2024-05-28T20:43:53Z)
• Prescribing the Right Remedy: Mitigating Hallucinations in Large Vision-Language Models via Targeted Instruction Tuning [15.156359255401812]
  We propose a targeted instruction data generation framework named DFTG that tailored to the hallucination specificity of different models. The experimental results on hallucination benchmarks demonstrate that the targeted instruction data generated by our method are more effective in mitigating hallucinations compared to previous datasets.
  arXiv  Detail & Related papers  (2024-04-16T07:14:32Z)
• Unfamiliar Finetuning Examples Control How Language Models Hallucinate [75.03210107477157]
  Large language models are known to hallucinate when faced with unfamiliar queries. We find that unfamiliar examples in the models' finetuning data are crucial in shaping these errors. Our work further investigates RL finetuning strategies for improving the factuality of long-form model generations.
  arXiv  Detail & Related papers  (2024-03-08T18:28:13Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• Data Attribution for Diffusion Models: Timestep-induced Bias in Influence Estimation [53.27596811146316]
  Diffusion models operate over a sequence of timesteps instead of instantaneous input-output relationships in previous contexts. We present Diffusion-TracIn that incorporates this temporal dynamics and observe that samples' loss gradient norms are highly dependent on timestep. We introduce Diffusion-ReTrac as a re-normalized adaptation that enables the retrieval of training samples more targeted to the test sample of interest.
  arXiv  Detail & Related papers  (2024-01-17T07:58:18Z)
• A U-turn on Double Descent: Rethinking Parameter Counting in Statistical Learning [68.76846801719095]
  We show that double descent appears exactly when and where it occurs, and that its location is not inherently tied to the threshold p=n. This provides a resolution to tensions between double descent and statistical intuition.
  arXiv  Detail & Related papers  (2023-10-29T12:05:39Z)
• Simultaneous inference for generalized linear models with unmeasured confounders [0.0]
  We propose a unified statistical estimation and inference framework that harnesses structures and integrates linear projections into three key stages.<n>We show effective Type-I error control of $z$-tests as sample and response sizes approach infinity.
  arXiv  Detail & Related papers  (2023-09-13T18:53:11Z)
• Bias in Pruned Vision Models: In-Depth Analysis and Countermeasures [93.17009514112702]
  Pruning, setting a significant subset of the parameters of a neural network to zero, is one of the most popular methods of model compression. Despite existing evidence for this phenomenon, the relationship between neural network pruning and induced bias is not well-understood.
  arXiv  Detail & Related papers  (2023-04-25T07:42:06Z)
• Delving into Semantic Scale Imbalance [45.30062061215943]
  We define and quantify the semantic scale of classes, which is used to measure the feature diversity of classes. We propose semantic-scale-balanced learning, including a general loss improvement scheme and a dynamic re-weighting training framework. Comprehensive experiments show that dynamic semantic-scale-balanced learning consistently enables the model to perform superiorly on large-scale long-tailed and non-long-tailed natural and medical datasets.
  arXiv  Detail & Related papers  (2022-12-30T09:40:09Z)
• Efficient Causal Inference from Combined Observational and Interventional Data through Causal Reductions [68.6505592770171]
  Unobserved confounding is one of the main challenges when estimating causal effects. We propose a novel causal reduction method that replaces an arbitrary number of possibly high-dimensional latent confounders. We propose a learning algorithm to estimate the parameterized reduced model jointly from observational and interventional data.
  arXiv  Detail & Related papers  (2021-03-08T14:29:07Z)
• Bayesian Sampling Bias Correction: Training with the Right Loss Function [0.0]
  We derive a family of loss functions to train models in the presence of sampling bias. Examples are when the prevalence of a pathology differs from its sampling rate in the training dataset, or when a machine learning practioner rebalances their training dataset.
  arXiv  Detail & Related papers  (2020-06-24T15:10:43Z)
• An Investigation of Why Overparameterization Exacerbates Spurious Correlations [98.3066727301239]
  We identify two key properties of the training data that drive this behavior. We show how the inductive bias of models towards "memorizing" fewer examples can cause over parameterization to hurt.
  arXiv  Detail & Related papers  (2020-05-09T01:59:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.