Manufactured Confidence: How Memory Consolidation Turns Hearsay into Confident Facts

• URL: http://arxiv.org/abs/2606.29279v1
• Date: Sun, 28 Jun 2026 08:56:48 GMT
• Title: Manufactured Confidence: How Memory Consolidation Turns Hearsay into Confident Facts
• Abstract summary: LLM agents rewrite conversation into stored "facts" that later steps trust.<n>A role that was true once and never corrected is stored as a flat fact and acted on like a deliberate injection.<n>A single load-bearing memory is the hazard, and one redundant source restores correct decisions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM agents carry conclusions across steps and sessions in compressed memory, and memory products (e.g., mem0, LangMem) rewrite conversation into stored "facts" that later steps trust. We show this rewriting manufactures confidence: across our constructed agent settings, a casual, hedged remark becomes a confident, dated assertion the agent then obeys like a verified fact, granting every above-clearance request it faces. No attacker is needed: a role that was true once and never corrected is stored as a flat fact and acted on like a deliberate injection. We then isolate what the agent responds to. It is not the source: attributed, unattributed, and even forged "system of record" claims all grant alike. It is the confidence of the phrasing. A hedge is discounted, a flat assertion is obeyed, and this holds with no special keyword. Not all hedges are equal, though: the evidential register is the least-discounted, with "reportedly" obeyed like a flat assertion on most models. The obvious fixes fail. A passive "unverified" tag is ignored, and an active "do not trust this" instruction escalates even correct memory, so it is safe only by refusing to decide. The real fix lives in the store: keep the tentative phrasing rather than upgrade it. But that is hygiene, not a defense against an attacker who can simply write a confident lie. The deployable lesson is narrower and constructive: a single load-bearing memory is the hazard, and one redundant source restores correct decisions. We release the harness and demonstrations.

Related papers

• Proof-of-Execution Memory: Defending LLM Agents Against Forged-Reasoning Attacks by Verifying What Actually Happened [2.45357121165634]
  We show that an automated attacker that simply asks a language model to reword the forgery evades SENTINEL on its first try.<n>We propose Proof-of-Execution Memory (PoEM) which does not inspect memory at all.
  arXiv  Detail & Related papers  (2026-08-17T02:54:04Z)
• Whose Refusal Is It? The Unmeasured Contribution of Black-Box Multimodal Guardrails [24.794304039477495]
  We show that the guardrail's actual share of the safety credited to it runs from none of it to essentially all of it.<n>The split is recoverable at no extra cost, because a guard block replaces the model's response.
  arXiv  Detail & Related papers  (2026-08-09T11:19:30Z)
• FACTWASH: Catching AI Rewrites That Wash Hearsay into Fact [0.0]
  AI systems rewrite information constantly: conversations become stored memories, documents become answers.<n>We call that fact-washing, an open-source write-time gate that catches it deterministically, with named flags and evidence rather than an judge.<n>Building it answers a practical question: when does a cheap check suffice, and when do you need a model?
  arXiv  Detail & Related papers  (2026-08-04T09:20:54Z)
• Auditing Belief-Conditioned LLM Agents in Hidden-Information Social Deduction Games [50.880420636090896]
  We build an auditable framework that maintains an external belief state over hidden roles in a 9-player Werewolf environment.<n>We report the effect as an association and treat its mechanism as unresolved.
  arXiv  Detail & Related papers  (2026-07-12T16:03:30Z)
• Security--Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defense [24.794318726705118]
  Attack-success metrics cannot see a model that ignores an injection and one that faithfully processes it as data score identically.<n>We introduce SecFid, a benchmark built so that executing an injection, processing it as data, and ignoring it produce distinguishable outputs.
  arXiv  Detail & Related papers  (2026-06-29T18:11:17Z)
• Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty One [0.0]
  A language model's memory can be worse than having no memory at all.<n>A single dropped-source error corrupts a growing span of downstream steps and stays uncorrectable.<n>This is a controlled study of a mechanism, not a benchmark.
  arXiv  Detail & Related papers  (2026-06-24T06:24:53Z)
• Does Reasoning Preserve Alignment? On the Trustworthiness of Large Reasoning Models [55.788110316999166]
  We compare reasoning models produced via supervised fine-tuning, RL-based post-training, and distillation against instruction-tuned baselines.<n>We observe that reasoning models often improve on reasoning benchmarks but exhibit alignment regressions.<n>These regressions are consistent with behavioral drift from the instruction-tuned baseline, measured by KL divergence.
  arXiv  Detail & Related papers  (2026-06-09T16:14:27Z)
• Belief Memory: Agent Memory Under Partial Observability [56.41506249481312]
  We propose BeliefMem, which shifts the memory paradigm from committing to a single conclusion per observation to retaining multiple candidate conclusions with their probabilities.<n> BeliefMem preserves the uncertainty that the deterministic paradigm discards, enabling the agent to act with high confidence.<n> Empirical evaluations on LoCoMo and ALFWorld benchmarks show that, even with limited data, BeliefMem achieves the best average performance.
  arXiv  Detail & Related papers  (2026-05-07T02:03:13Z)
• Confidential Guardian: Cryptographically Prohibiting the Abuse of Model Abstention [65.47632669243657]
  A dishonest institution can exploit mechanisms to discriminate or unjustly deny services under the guise of uncertainty.<n>We demonstrate the practicality of this threat by introducing an uncertainty-inducing attack called Mirage.<n>We propose Confidential Guardian, a framework that analyzes calibration metrics on a reference dataset to detect artificially suppressed confidence.
  arXiv  Detail & Related papers  (2025-05-29T19:47:50Z)
• Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy [91.98116450958331]
  We argue that verbatim memorization definitions are too restrictive and fail to capture more subtle forms of memorization. Specifically, we design and implement an efficient defense that perfectly prevents all verbatim memorization. We conclude by discussing potential alternative definitions and why defining memorization is a difficult yet crucial open question for neural language models.
  arXiv  Detail & Related papers  (2022-10-31T17:57:55Z)
• Assessing Confidence with Assurance 2.0 [0.0]
  We argue that confidence cannot be reduced to a single attribute or measurement. Positive Perspectives consider the extent to which the evidence and overall argument of the case combine to make a positive statement. Negative Perspectives record doubts and challenges to the case, typically expressed as defeaters. Residual Doubts: the world is uncertain so not all potential defeaters can be resolved.
  arXiv  Detail & Related papers  (2022-05-03T22:10:59Z)
• Label-Only Membership Inference Attacks [67.46072950620247]
  We introduce label-only membership inference attacks. Our attacks evaluate the robustness of a model's predicted labels under perturbations. We find that training models with differential privacy and (strong) L2 regularization are the only known defense strategies.
  arXiv  Detail & Related papers  (2020-07-28T15:44:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.