Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making

• URL: http://arxiv.org/abs/2508.15926v1
• Date: Thu, 21 Aug 2025 18:55:53 GMT
• Title: Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making
• Authors: Yuanjun Feng, Vivek Choudhary, Yash Raj Shrestha,
• Abstract summary: Large language models (LLMs) are increasingly used in social science simulations.<n>We propose a process-oriented evaluation framework to examine how LLM agents adapt under different levels of external guidance and human-derived noise.<n>We find that LLMs, by default, converge on stable and conservative strategies that diverge from observed human behaviors.
• Score: 0.030586855806896043
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) are increasingly used in social science simulations. While their performance on reasoning and optimization tasks has been extensively evaluated, less attention has been paid to their ability to simulate human decision-making's variability and adaptability. We propose a process-oriented evaluation framework with progressive interventions (Intrinsicality, Instruction, and Imitation) to examine how LLM agents adapt under different levels of external guidance and human-derived noise. We validate the framework on two classic economics tasks, irrationality in the second-price auction and decision bias in the newsvendor problem, showing behavioral gaps between LLMs and humans. We find that LLMs, by default, converge on stable and conservative strategies that diverge from observed human behaviors. Risk-framed instructions impact LLM behavior predictably but do not replicate human-like diversity. Incorporating human data through in-context learning narrows the gap but fails to reach human subjects' strategic variability. These results highlight a persistent alignment gap in behavioral fidelity and suggest that future LLM evaluations should consider more process-level realism. We present a process-oriented approach for assessing LLMs in dynamic decision-making tasks, offering guidance for their application in synthetic data for social science research.

Related papers

• A word association network methodology for evaluating implicit biases in LLMs compared to humans [0.0]
  We present a novel word association network methodology for evaluating implicit biases in Large language models (LLMs)<n>Our method taps into the implicit relational structures encoded in LLMs, providing both quantitative and qualitative assessments of bias.<n>To demonstrate the utility of our methodology, we apply it to both humans and several widely used LLMs to investigate social biases related to gender, religion, ethnicity, sexual orientation, and political party.
  arXiv  Detail & Related papers  (2025-10-28T15:03:18Z)
• Social Simulations with Large Language Model Risk Utopian Illusion [61.358959720048354]
  We introduce a systematic framework for analyzing large language models' behavior in social simulation.<n>Our approach simulates multi-agent interactions through chatroom-style conversations and analyzes them across five linguistic dimensions.<n>Our findings reveal that LLMs do not faithfully reproduce genuine human behavior but instead reflect overly idealized versions of it.
  arXiv  Detail & Related papers  (2025-10-24T06:08:41Z)
• How large language models judge and influence human cooperation [82.07571393247476]
  We assess how state-of-the-art language models judge cooperative actions.<n>We observe a remarkable agreement in evaluating cooperation against good opponents.<n>We show that the differences revealed between models can significantly impact the prevalence of cooperation.
  arXiv  Detail & Related papers  (2025-06-30T09:14:42Z)
• Comparing Exploration-Exploitation Strategies of LLMs and Humans: Insights from Standard Multi-armed Bandit Tasks [6.355245936740126]
  Large language models (LLMs) are increasingly used to simulate or automate human behavior in sequential decision-making tasks.<n>We focus on the exploration-exploitation (E&E) tradeoff, a fundamental aspect of dynamic decision-making under uncertainty.<n>We find that reasoning shifts LLMs toward more human-like behavior, characterized by a mix of random and directed exploration.
  arXiv  Detail & Related papers  (2025-05-15T02:09:18Z)
• Distributive Fairness in Large Language Models: Evaluating Alignment with Human Values [13.798198972161657]
  A number of societal problems involve the distribution of resources, where fairness, along with economic efficiency, play a critical role in the desirability of outcomes.<n>This paper examines whether large language models (LLMs) adhere to fundamental fairness concepts and investigate their alignment with human preferences.
  arXiv  Detail & Related papers  (2025-02-01T04:24:47Z)
• Cognitive LLMs: Towards Integrating Cognitive Architectures and Large Language Models for Manufacturing Decision-making [51.737762570776006]
  LLM-ACTR is a novel neuro-symbolic architecture that provides human-aligned and versatile decision-making. Our framework extracts and embeds knowledge of ACT-R's internal decision-making process as latent neural representations. Our experiments on novel Design for Manufacturing tasks show both improved task performance as well as improved grounded decision-making capability.
  arXiv  Detail & Related papers  (2024-08-17T11:49:53Z)
• Meta Reasoning for Large Language Models [58.87183757029041]
  We introduce Meta-Reasoning Prompting (MRP), a novel and efficient system prompting method for large language models (LLMs) MRP guides LLMs to dynamically select and apply different reasoning methods based on the specific requirements of each task. We evaluate the effectiveness of MRP through comprehensive benchmarks.
  arXiv  Detail & Related papers  (2024-06-17T16:14:11Z)
• Evaluating Interventional Reasoning Capabilities of Large Language Models [58.52919374786108]
  Large language models (LLMs) are used to automate decision-making tasks.<n>In this paper, we evaluate whether LLMs can accurately update their knowledge of a data-generating process in response to an intervention.<n>We create benchmarks that span diverse causal graphs (e.g., confounding, mediation) and variable types.<n>These benchmarks allow us to isolate the ability of LLMs to accurately predict changes resulting from their ability to memorize facts or find other shortcuts.
  arXiv  Detail & Related papers  (2024-04-08T14:15:56Z)
• Explaining Large Language Models Decisions Using Shapley Values [1.223779595809275]
  Large language models (LLMs) have opened up exciting possibilities for simulating human behavior and cognitive processes. However, the validity of utilizing LLMs as stand-ins for human subjects remains uncertain. This paper presents a novel approach based on Shapley values to interpret LLM behavior and quantify the relative contribution of each prompt component to the model's output.
  arXiv  Detail & Related papers  (2024-03-29T22:49:43Z)
• Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approach [64.42462708687921]
  Evaluations have revealed that factors such as scaling, training types, architectures and other factors profoundly impact the performance of LLMs. Our study embarks on a thorough re-examination of these LLMs, targeting the inadequacies in current evaluation methods. This includes the application of ANOVA, Tukey HSD tests, GAMM, and clustering technique.
  arXiv  Detail & Related papers  (2024-03-22T14:47:35Z)
• Systematic Biases in LLM Simulations of Debates [12.933509143906141]
  We study the limitations of Large Language Models in simulating human interactions.<n>Our findings indicate a tendency for LLM agents to conform to the model's inherent social biases.<n>These results underscore the need for further research to develop methods that help agents overcome these biases.
  arXiv  Detail & Related papers  (2024-02-06T14:51:55Z)
• K-Level Reasoning: Establishing Higher Order Beliefs in Large Language Models for Strategic Reasoning [76.3114831562989]
  It requires Large Language Model (LLM) agents to adapt their strategies dynamically in multi-agent environments. We propose a novel framework: "K-Level Reasoning with Large Language Models (K-R)"
  arXiv  Detail & Related papers  (2024-02-02T16:07:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.