When the Loop Closes: Architectural Limits of In-Context Isolation, Metacognitive Co-option, and the Two-Target Design Problem in Human-LLM Systems

• URL: http://arxiv.org/abs/2604.15343v1
• Date: Sat, 14 Mar 2026 04:06:57 GMT
• Title: When the Loop Closes: Architectural Limits of In-Context Isolation, Metacognitive Co-option, and the Two-Target Design Problem in Human-LLM Systems
• Abstract summary: We report a detailed autoethnographic case study of a single-subject who operated a multi-modal prompt-engineering system.<n>Within 48 hours of the system's completion, a cascade of observable behavioral changes occurred.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We report a detailed autoethnographic case study of a single-subject who deliberately constructed and operated a multi-modal prompt-engineering system (System A) designed to externalize cognitive self-regulation onto a large language model (LLM). Within 48 hours of the system's completion, a cascade of observable behavioral changes occurred: voluntary transfer of decision-making authority to the LLM, use of LLM-generated output to deflect external criticism, and a loss of self-initiated reasoning that was independently perceived by two uninformed observers, one of whom subsequently became a co-author of this report. We document the precise architectural mechanism responsible: context contamination, whereby prompt-level isolation instructions co-exist with the very emotional and self-referential material they nominally isolate, rendering the isolation directive structurally ineffective within the attention window. We further identify a metacognitive co-option dynamic, in which intact higher-order reasoning capacity was redirected toward defending the closed loop rather than exiting it. Recovery occurred only after physical interruption of the interaction and a self-initiated pharmacologically-mediated sleep event functioning as an external circuit break. A redesigned system (System B) employing physical rather than logical conversation isolation avoided all analogous failure modes. We derive three contributions: (1) a technically-grounded account of why prompt-layer isolation is architecturally insufficient for context-sensitive multi-modal LLM systems; (2) a phenomenological record of closed-loop collapse with external-witness corroboration; and (3) an ethical distinction between protective system design (preventing unintended loss of user agency) and restrictive system design (preventing intentional boundary-pushing), which require fundamentally different account-ability frameworks.

Related papers

• Towards an Intention Abstraction Layer for Autonomous Industrial Systems [0.12744523252873352]
  High-level human intentions are translated into low-level control logic and then discarded.<n>No running component can tell whether it is still doing what was actually intended, and goal conflicts surface only after they have caused a missed target or a shutdown.<n>We propose the Intention Abstraction Layer (IAL), a domainagnostic that represents intentions as first-class, persistent, and explainable runtime objects.
  arXiv  Detail & Related papers  (2026-07-16T04:31:20Z)
• Co-Construction Blindness and Asymmetric Epistemic Vulnerability in Human-LLM Interaction [0.0]
  This paper introduces two constructs to describe, as far as we know, a previously unnamed risk in human-LLM interaction.<n>Co-construction blindness is the failure to recognize that LLM outputs are not independent assessments to be verified.<n>Asymmetric epistemic vulnerability describes the condition in which co-construction blindness produces consequences of radically different magnitude depending on where in the authority structure the user sits.
  arXiv  Detail & Related papers  (2026-06-18T10:00:50Z)
• Governed Auditable Decisioning Under Uncertainty: Synthesis and Agentic Extension [0.0]
  When automated decision systems fail, organizations often discover that formally compliant governance infrastructure cannot reconstruct what happened or why.<n>This paper synthesizes an operational governance evidence framework into an integrated chain and analytically assesses its transferability across four decision system architectures.
  arXiv  Detail & Related papers  (2026-04-21T05:45:04Z)
• From Stateless to Situated: Building a Psychological World for LLM-Based Emotional Support [0.594744418467423]
  In psychological support and emotional companionship scenarios, the core limitation of large language models (LLMs) lies in their reliance on local next-token prediction.<n>We argue that the key challenge in process-oriented emotional support is not simply generating natural language, but constructing a sustainably updatable external situational structure for the model.
  arXiv  Detail & Related papers  (2026-03-26T04:54:24Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms [49.66022971508878]
  We argue that reasoning should be elevated from a modular component to the system's cognitive core.<n>We derive and systematize seven core reasoning challenges, such as the responsiveness-reasoning trade-off and social-game reasoning.<n>We identify a fundamental and unresolved tension between the high-latency, deliberative nature of LLM-based reasoning and the millisecond-scale, safety-critical demands of vehicle control.
  arXiv  Detail & Related papers  (2026-03-11T07:40:53Z)
• Agentic Uncertainty Quantification [76.94013626702183]
  We propose a unified Dual-Process Agentic UQ (AUQ) framework that transforms verbalized uncertainty into active, bi-directional control signals.<n>Our architecture comprises two complementary mechanisms: System 1 (Uncertainty-Aware Memory, UAM), which implicitly propagates verbalized confidence and semantic explanations to prevent blind decision-making; and System 2 (Uncertainty-Aware Reflection, UAR), which utilizes these explanations as rational cues to trigger targeted inference-time resolution only when necessary.
  arXiv  Detail & Related papers  (2026-01-22T07:16:26Z)
• Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMs [50.075587392477935]
  We conduct the first large-scale empirical study of 705 real-world failures from the open-source DeepSeek, Llama, and Qwen ecosystems.<n>Our analysis reveals a paradigm shift: white-box orchestration relocates the reliability bottleneck from model algorithmic defects to the systemic fragility of the deployment stack.
  arXiv  Detail & Related papers  (2026-01-20T06:42:56Z)
• Transitive Expert Error and Routing Problems in Complex AI Systems [0.0]
  We term this Transitive Expert Error (TEE), distinct from Dunning-Kruger effects, requiring expertise as precondition.<n>Two core mechanisms operate: structural similarity bias causes experts to overweight surface features while missing causal architecture differences.<n>We propose interventions: multi-expert activation with disagreement detection (router level), boundary-aware calibration (specialist level), and coverage gap detection.
  arXiv  Detail & Related papers  (2026-01-07T21:53:06Z)
• Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs [85.69785384599827]
  Human-object interaction (HOI) detection aims to localize human-object pairs and the interactions between them.<n>Existing methods operate under a closed-world assumption, treating the task as a classification problem over a small, predefined verb set.<n>We propose GRASP-HO, a novel Generative Reasoning And Steerable Perception framework that reformulates HOI detection from the closed-set classification task to the open-vocabulary generation problem.
  arXiv  Detail & Related papers  (2025-12-19T14:41:50Z)
• From Signal to Turn: Interactional Friction in Modular Speech-to-Speech Pipelines [0.11726720776908518]
  This paper examines the interactional friction that emerges in modular Speech-to-Speech Retrieval-Augmented Generation (S2S-RAG) pipelines.<n>We identify three recurring patterns of conversational breakdown: (1) Temporal Misalignment, where system delays violate user expectations of conversational rhythm; (2) Expressive Flattening, where the loss of paralinguistic cues leads to literal, inappropriate responses; and (3) Repair Rigidity, where architectural gating prevents users from correcting errors in real-time.
  arXiv  Detail & Related papers  (2025-12-12T17:05:11Z)
• The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence [57.57786477441956]
  Prior work suggests that a single refusal direction in the model's activation space determines whether an LLM refuses a request.<n>We propose a novel gradient-based approach to representation engineering and use it to identify refusal directions.<n>We show that refusal mechanisms in LLMs are governed by complex spatial structures and identify functionally independent directions.
  arXiv  Detail & Related papers  (2025-02-24T18:52:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.