Large Language Model Recall Uncertainty is Modulated by the Fan Effect
- URL: http://arxiv.org/abs/2407.06349v1
- Date: Mon, 8 Jul 2024 19:40:50 GMT
- Title: Large Language Model Recall Uncertainty is Modulated by the Fan Effect
- Authors: Jesse Roberts, Kyle Moore, Thao Pham, Oseremhen Ewaleifoh, Doug Fisher,
- Abstract summary: This paper evaluates whether large language models (LLMs) exhibit cognitive fan effects, similar to those discovered by Anderson in humans.
We conduct two sets of in-context recall experiments designed to elicit fan effects.
- Score: 0.0
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: This paper evaluates whether large language models (LLMs) exhibit cognitive fan effects, similar to those discovered by Anderson in humans, after being pre-trained on human textual data. We conduct two sets of in-context recall experiments designed to elicit fan effects. Consistent with human results, we find that LLM recall uncertainty, measured via token probability, is influenced by the fan effect. Our results show that removing uncertainty disrupts the observed effect. The experiments suggest the fan effect is consistent whether the fan value is induced in-context or in the pre-training data. Finally, these findings provide in-silico evidence that fan effects and typicality are expressions of the same phenomena.
Related papers
- Unfamiliar Finetuning Examples Control How Language Models Hallucinate [75.03210107477157]
Large language models are known to hallucinate when faced with unfamiliar queries.
We find that unfamiliar examples in the models' finetuning data are crucial in shaping these errors.
Our work further investigates RL finetuning strategies for improving the factuality of long-form model generations.
arXiv Detail & Related papers (2024-03-08T18:28:13Z) - Decoding Susceptibility: Modeling Misbelief to Misinformation Through a
Computational Approach [63.67533153887132]
Susceptibility to misinformation describes the degree of belief in unverifiable claims that is not observable.
Existing susceptibility studies heavily rely on self-reported beliefs.
We propose a computational approach to model users' latent susceptibility levels.
arXiv Detail & Related papers (2023-11-16T07:22:56Z) - Stubborn Lexical Bias in Data and Models [50.79738900885665]
We use a new statistical method to examine whether spurious patterns in data appear in models trained on the data.
We apply an optimization approach to *reweight* the training data, reducing thousands of spurious correlations.
Surprisingly, though this method can successfully reduce lexical biases in the training data, we still find strong evidence of corresponding bias in the trained models.
arXiv Detail & Related papers (2023-06-03T20:12:27Z) - Do Large Language Models Show Decision Heuristics Similar to Humans? A
Case Study Using GPT-3.5 [0.0]
GPT-3.5 is an example of an LLM that supports a conversational agent called ChatGPT.
In this work, we used a series of novel prompts to determine whether ChatGPT shows biases, and other decision effects.
We also tested the same prompts on human participants.
arXiv Detail & Related papers (2023-05-08T01:02:52Z) - Linking a predictive model to causal effect estimation [21.869233469885856]
This paper first tackles the challenge of estimating the causal effect of any feature (as the treatment) on the outcome w.r.t. a given instance.
The theoretical results naturally link a predictive model to causal effect estimations and imply that a predictive model is causally interpretable.
We use experiments to demonstrate that various types of predictive models, when satisfying the conditions identified in this paper, can estimate the causal effects of features as accurately as state-of-the-art causal effect estimation methods.
arXiv Detail & Related papers (2023-04-10T13:08:16Z) - Susceptibility to Influence of Large Language Models [5.931099001882958]
Two studies tested the hypothesis that a Large Language Model (LLM) can be used to model psychological change following exposure to influential input.
The first study tested a generic mode of influence - the Illusory Truth Effect (ITE)
The second study concerns a specific mode of influence - populist framing of news to increase its persuasion and political mobilization.
arXiv Detail & Related papers (2023-03-10T16:53:30Z) - CausalDialogue: Modeling Utterance-level Causality in Conversations [83.03604651485327]
We have compiled and expanded upon a new dataset called CausalDialogue through crowd-sourcing.
This dataset includes multiple cause-effect pairs within a directed acyclic graph (DAG) structure.
We propose a causality-enhanced method called Exponential Average Treatment Effect (ExMATE) to enhance the impact of causality at the utterance level in training neural conversation models.
arXiv Detail & Related papers (2022-12-20T18:31:50Z) - Identifying Peer Influence in Therapeutic Communities Adjusting for Latent Homophily [1.6385815610837167]
We investigate peer role model influence on successful graduation from Therapeutic Communities (TCs) for substance abuse and criminal behavior.
To identify peer influence in the presence of unobserved homophily in observational data, we model the network with a latent variable model.
Our results indicate a positive effect of peers' graduation on residents' graduation and that it differs based on gender, race, and the definition of the role model effect.
arXiv Detail & Related papers (2022-03-27T06:47:28Z) - Causal Effect Estimation using Variational Information Bottleneck [19.6760527269791]
Causal inference is to estimate the causal effect in a causal relationship when intervention is applied.
We propose a method to estimate Causal Effect by using Variational Information Bottleneck (CEVIB)
arXiv Detail & Related papers (2021-10-26T13:46:12Z) - Efficient Causal Inference from Combined Observational and
Interventional Data through Causal Reductions [68.6505592770171]
Unobserved confounding is one of the main challenges when estimating causal effects.
We propose a novel causal reduction method that replaces an arbitrary number of possibly high-dimensional latent confounders.
We propose a learning algorithm to estimate the parameterized reduced model jointly from observational and interventional data.
arXiv Detail & Related papers (2021-03-08T14:29:07Z) - CausalVAE: Structured Causal Disentanglement in Variational Autoencoder [52.139696854386976]
The framework of variational autoencoder (VAE) is commonly used to disentangle independent factors from observations.
We propose a new VAE based framework named CausalVAE, which includes a Causal Layer to transform independent factors into causal endogenous ones.
Results show that the causal representations learned by CausalVAE are semantically interpretable, and their causal relationship as a Directed Acyclic Graph (DAG) is identified with good accuracy.
arXiv Detail & Related papers (2020-04-18T20:09:34Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.