On the Inseparability of Instructions and Data in Shared-Embedding Sequence Models

• URL: http://arxiv.org/abs/2606.27567v1
• Date: Thu, 25 Jun 2026 21:52:29 GMT
• Title: On the Inseparability of Instructions and Data in Shared-Embedding Sequence Models
• Abstract summary: We formalize prompted systems as Prompted Action Models with control-authoritative actions.<n>We prove Semantic-Faithful Control (SFC) is unachievable within the shared pipeline.<n>We identify the root cause and the class of solution it demands.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt injection is the top security risk for LLM-integrated applications, yet every defense proposed so far has been broken. We prove this is not a coincidence: in shared-embedding architectures that lack enforced control-data separation, perfect prompt-injection prevention is mathematically impossible. We formalize prompted systems as Prompted Action Models whose outputs include control-authoritative actions: refusal decisions, tool authorization, policy routing, and memory writes. We define Semantic-Faithful Control (SFC), the property that such behavior depends only on the meaning of untrusted input, not on how it is encoded. We then prove SFC is unachievable within the shared pipeline, via three results: a provenance-recovery impossibility (shared representations make trusted and untrusted content statistically inseparable, bounded by total variation distance); control-path exposure (untrusted tokens enter control-relevant computation through the same attention value-aggregation that determines outputs); and a finite-coverage invariance gap (finite training cannot certify invariance over infinite semantic-equivalence classes). We ground each quantity in measurements on production tokenizers and models. The result is structural, not a gap in current defenses. It mirrors the code-data confusion in Von Neumann machines that gives rise to buffer overflows, a vulnerability class that took decades of layered defenses (DEP, Write-XOR-Execute, ASLR, stack canaries, and ultimately memory-safe languages) to contain, because no single mechanism sufficed. The implication is the same: prompt injection cannot be eliminated by better in-pipeline classification or alignment alone. It requires architectural separation of instruction and data channels. We identify the root cause and the class of solution it demands.

Related papers

• Recognition Without Enforcement: Configuration-Dependent Failures in LLM Agent Instruction Arbitration and External Control [0.0]
  LLM agents arbitrate among instructions from system prompts, users, memory, and tools, but this arbitration cannot be assumed to enforce trust boundaries.<n>We identify a recognition-enforcement gap: source-format features are linearly decodable from model activations, and models can explicitly identify forged authority when prompted.<n>We treat model self-arbitration as a capability rather than a security boundary and implement an external reference monitor combining authenticated source routing with capability-gated tool execution.
  arXiv  Detail & Related papers  (2026-08-28T16:34:05Z)
• LMSM: LLM Security Framework Inspired by Linux Security Modules [56.93644694627783]
  Large language models (LLMs) are increasingly deployed with layered defenses, yet malicious prompts can still bypass them.<n>We present Language Model Security Modules (LMSM), a security framework that adapts the separation behind Linux Security Modules (LSM) to LLM serving.
  arXiv  Detail & Related papers  (2026-08-26T12:13:05Z)
• AI Slop and Hallucinations in Vulnerability Assessment: A Survey on Reasoning Failures and Trustworthy Mitigation [48.55515767840701]
  "AI slop" is artifacts, hallucinated vulnerabilities, plausible but incorrect patches, and semantically repackaged bug reports.<n>This paper surveys the empirical evidence, identifies a unifying mechanism, and traces a path toward trustworthy triage.
  arXiv  Detail & Related papers  (2026-08-26T11:48:38Z)
• Certifying Plans under Model Mismatch: A Trilemma for Reachability from Scarce Data [5.455772242709494]
  Sim-to-real policies are designed under nominal dynamics, but target-system trials may yield only a few isolated one-step transitions.<n>We study pre-execution certification of a fixed control sequence, such as an action chunk produced by a learned policy.
  arXiv  Detail & Related papers  (2026-08-03T16:28:26Z)
• Fidelity Is Not Safety: Gently-Compressed LLMs Pass Every Data-Free Quality Guard Yet Invent Procedure Steps in Agentic Execution [0.0]
  Practitioners accept a compressed language model once it clears a stack of data-cheap quality guards.<n>Across three model families, gently-compressed models clear every guard and then invent procedure steps that were never in the instructions.<n>We characterize the blindspot and dissociation with paired confidence intervals on a pre-registered, powered canary.
  arXiv  Detail & Related papers  (2026-07-30T13:33:09Z)
• Instruction Bleed: Cross-Module Interference in Prompt-Composed Agentic Systems [4.22293816564147]
  Practitioners of prompt-composed agentic systems report a recurring failure mode: editing one prompt module silently shifts the behavior of others despite no shared variable or executable dependency.<n>We formalize this as compositional behavioral leakage (CBL): interference between modules sharing a context window.<n>We probe CBL on a deployed job-evaluation agent through a reusable three-channel protocol that perturbs paired non-focal modules along volume, content, and form.
  arXiv  Detail & Related papers  (2026-06-24T20:09:28Z)
• What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility [48.10132234701036]
  We propose a diagnostic evaluation protocol based on three checks -- scale stability, reference validity, and depth utility -- together with per-layer rather than aggregate cosine reporting.<n>Across multiple architectures and methods, our protocol identifies all failures with wide calibration margins.
  arXiv  Detail & Related papers  (2026-06-19T06:04:17Z)
• Bit-Exact AI Inference Verification Without Performance Tradeoffs [0.0]
  Covert adversaries can exploit unverifiable degrees of freedom in monitored computation.<n>We analyze how modern inference engines produce deterministic but non-invariant outputs.<n>We demonstrate that bitwise-precise re-computation does not require access to identical hardware.
  arXiv  Detail & Related papers  (2026-05-29T19:15:15Z)
• AgentSecBench: Measuring Prompt Injection, Privacy Leakage, and Tool-Use Integrity in LLM Agents [0.2864713389096699]
  We introduce AgentSecBench as an empirical instantiation of a formal security framework for this problem.<n>It defines three games-instruction-integrity, retrieval-confidentiality, and capability-integrity-under a common notion of intent-to-execution noninterference with permitted leakage.<n>It represents an application policy as a projection onto authorized observations and capabilities, distinguishes prompt annotations from enforcing projections, and measures both adversarial advantage and whether a defense closes the relevant model-visible channel before generation.
  arXiv  Detail & Related papers  (2026-05-25T18:53:22Z)
• Compiling Activation Steering into Weights via Null-Space Constraints for Stealthy Backdoors [48.881343993730844]
  Safety-aligned large language models (LLMs) are increasingly deployed in real-world pipelines.<n> adversaries can distribute backdoored checkpoints that behave normally under standard evaluation but jailbreak when a hidden trigger is present.<n>Recent post-hoc weight-editing methods offer an efficient approach to injecting such backdoors.
  arXiv  Detail & Related papers  (2026-04-14T06:48:33Z)
• Can AI Keep a Secret? Contextual Integrity Verification: A Provable Security Architecture for LLMs [0.0]
  We present Contextual Integrity Verification (CIV), a provenance-time security architecture that attaches cryptographic inferenceally signed labels to every token.<n>CIV provides provenance, per-token non-interference guarantees on frozen models.<n>We demonstrate drop-in protection for Llama-3-8B and Mistral-7B.
  arXiv  Detail & Related papers  (2025-08-12T18:47:30Z)
• Can LLMs Separate Instructions From Data? And What Do We Even Mean By That? [60.50127555651554]
  Large Language Models (LLMs) show impressive results in numerous practical applications, but they lack essential safety features.<n>This makes them vulnerable to manipulations such as indirect prompt injections and generally unsuitable for safety-critical tasks.<n>We introduce a formal measure for instruction-data separation and an empirical variant that is calculable from a model's outputs.
  arXiv  Detail & Related papers  (2024-03-11T15:48:56Z)
• Unsupervised Continual Anomaly Detection with Contrastively-learned Prompt [80.43623986759691]
  We introduce a novel Unsupervised Continual Anomaly Detection framework called UCAD. The framework equips the UAD with continual learning capability through contrastively-learned prompts. We conduct comprehensive experiments and set the benchmark on unsupervised continual anomaly detection and segmentation.
  arXiv  Detail & Related papers  (2024-01-02T03:37:11Z)
• Bilateral Dependency Optimization: Defending Against Model-inversion Attacks [61.78426165008083]
  We propose a bilateral dependency optimization (BiDO) strategy to defend against model-inversion attacks. BiDO achieves the state-of-the-art defense performance for a variety of datasets, classifiers, and MI attacks.
  arXiv  Detail & Related papers  (2022-06-11T10:07:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.