Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories

• URL: http://arxiv.org/abs/2606.04778v1
• Date: Wed, 03 Jun 2026 12:01:27 GMT
• Title: Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories
• Abstract summary: Safety-aligned Large Language Models (LLMs) remain vulnerable to interventions during inference that redirect generation toward harmful outputs.<n>Recent work attributes this to shallow safety, where alignment concentrates in the first few output tokens.<n>We show that short token injections at any generation step can substantially alter subsequent safety behavior.
• Score: 9.386028796990399
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Safety-aligned Large Language Models (LLMs) remain vulnerable to interventions during inference that redirect generation toward harmful outputs. Recent work attributes this to shallow safety, where alignment concentrates in the first few output tokens. We show that shallow safety is a special case of a broader inference-time vulnerability, in which short token injections at any generation step can substantially alter subsequent safety behavior. We also find that a model's alignment with refusal directions in its hidden states does not predict its robustness to such injection, revealing that internal state alone does not determine generation behavior under perturbation. To address this, we align models directly on generation trajectories constructed by simulating mid-sequence perturbation, and show that this improves robustness to mid-sequence injection and generalizes to attacks that exploit early-token generation. Our work argues that robust safety alignment requires training on the generation process itself, not only its outputs.

Related papers

• Do Models Share Safety Representations? Cross-Model Steering for Safe Visual Generation [52.122731171289665]
  We introduce the first framework for cross-model safety steering.<n>Our pipeline never accesses unsafe data on the target side.<n>We evaluate our approach in text-to-image and text-to-video generation across diverse source-target model pairs.
  arXiv  Detail & Related papers  (2026-06-03T18:00:04Z)
• When Autoregressive Consistency Hurts Safety Alignment [20.12397699480725]
  We show that autoregressive consistency can concentrate alignment updates on early tokens.<n>The same mechanism also predicts a broader class of attacks on large language models.<n>We propose adversarial safety alignment, an initial framework based on worst-case harmful continuation states.
  arXiv  Detail & Related papers  (2026-06-02T19:32:23Z)
• Selective Safety Steering via Value-Filtered Decoding [54.87935112120107]
  Large language models (LLMs) are trained to align with human values, but their generations may still violate safety constraints.<n>Existing decoding-time steering methods often intervene unnecessarily, modifying generations that would have been safe under the base model.<n>We propose a new test-time steering method designed to reduce such unnecessary interventions while improving the safety of unsafe responses.
  arXiv  Detail & Related papers  (2026-05-14T12:13:08Z)
• Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models [62.16655896700062]
  Activation steering is a technique to enhance the utility of Large Language Models (LLMs)<n>We show that it unintentionally introduces critical and under-explored safety risks.<n>Experiments reveal that these interventions act as a force multiplier, creating new vulnerabilities to jailbreaks and increasing attack success rates to over 80% on standard benchmarks.
  arXiv  Detail & Related papers  (2026-02-03T12:32:35Z)
• Self-Guard: Defending Large Reasoning Models via enhanced self-reflection [54.775612141528164]
  Self-Guard is a lightweight safety defense framework for Large Reasoning Models.<n>It bridges the awareness-compliance gap, achieving robust safety performance without compromising model utility.<n>Self-Guard exhibits strong generalization across diverse unseen risks and varying model scales.
  arXiv  Detail & Related papers  (2026-01-31T13:06:11Z)
• Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures [70.48661957773449]
  Emergent Misalignment refers to a failure mode in which fine-tuning large language models on narrowly scoped data induces broadly misaligned behavior.<n>Across multiple domains and model families, we find that fine-tuning models on data exhibiting specific character-level dispositions induces substantially stronger and more transferable misalignment than incorrect-advice fine-tuning.
  arXiv  Detail & Related papers  (2026-01-30T15:28:42Z)
• When Should We Introduce Safety Interventions During Pretraining? [100.3502954292386]
  Prior work has shown that interventions applied during pretraining, such as rephrasing harmful content, can substantially improve the safety of the resulting models.<n>We find that introducing interventions earlier generally yields more robust models with no increase in overrefusal rates.<n>We also see clear benefits in the steerability of models towards safer generations.
  arXiv  Detail & Related papers  (2026-01-11T22:38:17Z)
• Building a Foundational Guardrail for General Agentic Systems via Synthetic Data [76.18834864749606]
  LLM agents can plan multi-step tasks, intervening at the planning stage-before any action is executed-is often the safest way to prevent harm.<n>Existing guardrails mostly operate post-execution, which is difficult to scale and leaves little room for controllable supervision at the plan level.<n>We introduce AuraGen, a controllable engine that synthesizes benign trajectories, injects category-labeled risks with difficulty, and filters outputs via an automated reward model.
  arXiv  Detail & Related papers  (2025-10-10T18:42:32Z)
• Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift [23.0914017433021]
  This work identifies a novel class of deployment phase attacks that exploit a vulnerability by injecting imperceptible perturbations directly into the embedding layer outputs without modifying model weights or input text.<n>We propose Search based Embedding Poisoning, a practical, model agnostic framework that introduces carefully optimized perturbations into embeddings associated with high risk tokens.
  arXiv  Detail & Related papers  (2025-09-08T05:00:58Z)
• Probing the Robustness of Large Language Models Safety to Latent Perturbations [30.16804362984161]
  Safety alignment is a key requirement for building reliable Artificial General Intelligence.<n>We observe that minor latent shifts can still trigger unsafe responses in aligned models.<n>We introduce Layer-wise Adversarial Patch Training(LAPT), a fine-tuning strategy that injects controlled perturbations into hidden representations during training.
  arXiv  Detail & Related papers  (2025-06-19T07:03:05Z)
• Backtracking for Safety [11.141166381133054]
  Large language models (LLMs) have demonstrated remarkable capabilities across various tasks, but ensuring their safety and alignment with human values remains crucial.<n>Current safety alignment methods, such as supervised fine-tuning and reinforcement learning-based approaches, can exhibit vulnerabilities to adversarial attacks.<n>We propose a novel backtracking method designed to address these limitations.
  arXiv  Detail & Related papers  (2025-03-11T22:04:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.