BeliefShift: Benchmarking Temporal Belief Consistency and Opinion Drift in LLM Agents

• URL: http://arxiv.org/abs/2603.23848v1
• Date: Wed, 25 Mar 2026 02:09:35 GMT
• Title: BeliefShift: Benchmarking Temporal Belief Consistency and Opinion Drift in LLM Agents
• Abstract summary: BeliefShift is a benchmark designed to evaluate belief dynamics in multi-session conversational agents.<n>It covers three tracks: Temporal Belief Consistency, Contradiction Detection, and Evidence-Driven Revision.<n>We evaluate seven models including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, LLaMA-3, and Mistral-Large under zero-shot and retrieval-augmented generation settings.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLMs are increasingly used as long-running conversational agents, yet every major benchmark evaluating their memory treats user information as static facts to be stored and retrieved. That's the wrong model. People change their minds, and over extended interactions, phenomena like opinion drift, over-alignment, and confirmation bias start to matter a lot. BeliefShift introduces a longitudinal benchmark designed specifically to evaluate belief dynamics in multi-session LLM interactions. It covers three tracks: Temporal Belief Consistency, Contradiction Detection, and Evidence-Driven Revision. The dataset includes 2,400 human-annotated multi-session interaction trajectories spanning health, politics, personal values, and product preferences. We evaluate seven models including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, LLaMA-3, and Mistral-Large under zero-shot and retrieval-augmented generation (RAG) settings. Results reveal a clear trade-off: models that personalize aggressively resist drift poorly, while factually grounded models miss legitimate belief updates. We further introduce four novel evaluation metrics: Belief Revision Accuracy (BRA), Drift Coherence Score (DCS), Contradiction Resolution Rate (CRR), and Evidence Sensitivity Index (ESI).

Related papers

• Before You Poll with LLMs: A Deliberative Diagnostic Framework [0.2578242050187029]
  Current evaluations only test whether personas hold the right opinions -- a static snapshot.<n>We introduce the Deliberative Polling Diagnostic Framework, which compares human and LLM belief shifts after identical informational interventions.<n>Applying the framework to five frontier models using data from America in One Room, we find that every model fails.
  arXiv  Detail & Related papers  (2026-09-14T16:41:48Z)
• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality? [74.69723255239663]
  Grounded Personality Reasoning requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding.<n> MM-OCEAN is produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs.<n>The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%.
  arXiv  Detail & Related papers  (2026-05-21T07:42:47Z)
• AffectVerse: Emotional World Models for Multimodal Affective Computing [56.144242722718985]
  We propose AffectVerse, an action-free representation-level module for short-horizon latent affective prediction.<n>EWM contains three modules: 1) Cross-Modal Temporal Imagination predicts future video/audio representations from past tokens with multi-step rollout.<n>EWM compresses imagined tokens into modality-aware belief tokens.<n>These results suggest predictive belief-state modeling is a practical alternative for affective computing.
  arXiv  Detail & Related papers  (2026-05-19T15:05:00Z)
• Just Keep Prompting: Evaluating Repetitive Socratic Prompting in VLMs [7.833541893955439]
  We introduce Just Keep Prompting (JKP), a multi-turn evaluation framework that measures VLM stability when users repeatedly challenge, question, or contradict a model's answer.<n>JKP probes models for up to 10 follow-up turns using three strategies: Adversarial Negation (repeated rejection), Pure Socratic Interrogation (repeated calls to reassess certainty), and Context-Aware Socratic Summarization.<n>We evaluate GPT-4o, Gemini 2.5 Pro, and Qwen3-VL-30B on a subset of the STAR benchmark across 720 multi-turn runs.
  arXiv  Detail & Related papers  (2026-05-01T17:02:19Z)
• Dynamic Theory of Mind as a Temporal Memory Problem: Evidence from Large Language Models [0.8666275811953881]
  Theory of Mind (ToM) is central to social cognition and human-AI interaction.<n>Large Language Models (LLMs) have been used to help understand and represent ToM.<n>We introduce DToM-Track to investigate temporal belief reasoning in controlled multiturn conversations.
  arXiv  Detail & Related papers  (2026-03-15T22:54:03Z)
• Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge [0.0]
  This study systematically evaluates scoring stability across five commonly used models, GPT-4o, GPT-4o-mini, Gemini-2.5-Flash, Claude-Haiku-4.5, and Claude-Sonnet-4.5.<n>We address three questions: how stable a model's scores are across repeated runs, how differently models score identical inputs, and how temperature affects scoring consistency.
  arXiv  Detail & Related papers  (2026-02-04T16:03:08Z)
• Incoherent Beliefs & Inconsistent Actions in Large Language Models [33.54139088666698]
  Real-world tasks and environments exhibit differences from the static datasets that large language models (LLMs) are typically evaluated on.<n>We examine two critical components of LLM performance: the ability of LLMs to coherently update their beliefs, and the extent to which the actions they take are consistent with those beliefs.<n>Our results highlight the difficulties of predicting LLM behavior in complex real-world settings.
  arXiv  Detail & Related papers  (2025-11-17T11:04:00Z)
• Consistently Simulating Human Personas with Multi-Turn Reinforcement Learning [52.07170679746533]
  Large Language Models (LLMs) are increasingly used to simulate human users in interactive settings such as therapy, education, and social role-play.<n>We introduce a unified framework for evaluating and improving persona consistency in LLM-generated dialogue.<n>We define three automatic metrics: prompt-to-line consistency, line-to-line consistency, and Q&A consistency, that capture different types of persona drift and validate each against human annotations.
  arXiv  Detail & Related papers  (2025-10-31T19:40:41Z)
• Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL [64.3268313484078]
  Large Language Models (LLMs) interact with millions of people worldwide in applications such as customer support, education and healthcare.<n>Their ability to produce deceptive outputs, whether intentionally or inadvertently, poses significant safety concerns.<n>We investigate the extent to which LLMs engage in deception within dialogue, and propose the belief misalignment metric to quantify deception.
  arXiv  Detail & Related papers  (2025-10-16T05:29:36Z)
• Deliberative Dynamics and Value Alignment in LLM Debates [0.0]
  We examine deliberative dynamics and value alignment in multi-turn settings using large language models.<n>We test order effects and verdict revision in 1,000 dilemmas from Reddit's "Am I the Asshole" community.
  arXiv  Detail & Related papers  (2025-10-11T04:06:07Z)
• Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models [86.88657425848547]
  Large reasoning models (LRMs) already possess a latent capacity for long chain-of-thought reasoning.<n>We explicitly align models with three meta-abilities: deduction, induction, and abduction, using automatically generated, self-verifiable tasks.<n>Our three stage-pipeline individual alignment, parameter-space merging, and domain-specific reinforcement learning, boosts performance by over 10% relative to instruction-tuned baselines.
  arXiv  Detail & Related papers  (2025-05-15T17:58:33Z)
• If an LLM Were a Character, Would It Know Its Own Story? Evaluating Lifelong Learning in LLMs [55.8331366739144]
  We introduce LIFESTATE-BENCH, a benchmark designed to assess lifelong learning in large language models (LLMs)<n>Our fact checking evaluation probes models' self-awareness, episodic memory retrieval, and relationship tracking, across both parametric and non-parametric approaches.
  arXiv  Detail & Related papers  (2025-03-30T16:50:57Z)
• Belief Revision: The Adaptability of Large Language Models Reasoning [63.0281286287648]
  We introduce Belief-R, a new dataset designed to test LMs' belief revision ability when presented with new evidence. Inspired by how humans suppress prior inferences, this task assesses LMs within the newly proposed delta reasoning framework. We evaluate $sim$30 LMs across diverse prompting strategies and found that LMs generally struggle to appropriately revise their beliefs in response to new information.
  arXiv  Detail & Related papers  (2024-06-28T09:09:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.