When Autoregressive Consistency Hurts Safety Alignment

• URL: http://arxiv.org/abs/2606.04168v1
• Date: Tue, 02 Jun 2026 19:32:23 GMT
• Title: When Autoregressive Consistency Hurts Safety Alignment
• Abstract summary: We show that autoregressive consistency can concentrate alignment updates on early tokens.<n>The same mechanism also predicts a broader class of attacks on large language models.<n>We propose adversarial safety alignment, an initial framework based on worst-case harmful continuation states.
• Score: 20.12397699480725
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Safety alignment in large language models (LLMs) is fragile in part because it is often shallow: fine-tuning mainly reshapes the model's behavior near the first few output tokens. We argue that this phenomenon can be understood through autoregressive consistency, the tendency of next-token prediction to preserve and extend the current response trajectory consistently. By analyzing the learning dynamics of safety alignment, we show that autoregressive consistency can concentrate alignment updates on early tokens, offering a mechanistic explanation for shallow safety alignment. The same mechanism also predicts a broader class of attacks on LLMs: attacks that induce harmful continuation states at arbitrary positions in the output trajectory. As a concrete example, we introduce random insertion attack, which inserts a short harmful span into an otherwise safe refusal trajectory and exploits autoregressive consistency to sustain the resulting harmful branch, thereby bypassing safety alignment. Notably, a short harmful span can redirect the generation to be harmful even after a long refusal prefix, highlighting autoregressive consistency as a potential broader failure mechanism. This suggests that safety alignment should also break harmful autoregressive consistency throughout the output trajectory. We therefore propose adversarial safety alignment, an initial framework based on worst-case harmful continuation states, and instantiate it with random worst-insertion training. Overall, our results suggest that autoregressive consistency should be treated as a central consideration in both safety alignment and attack design.

Related papers

• When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning [19.41242962849505]
  We show that safety Fisher is low-rank, and alignment makes the safety geometry flatter while preserving an output-routing pathway.<n>LoRA and ASAM mitigate early collapse by suppressing output-side sharpness, but their protection weakens at larger fine-tuning scales.<n>Overall, safety failure is best understood as a disruption of a low-rank output-routing mechanism.
  arXiv  Detail & Related papers  (2026-09-01T15:59:32Z)
• HARC: Coupling Harmfulness and Refusal Directions for Robust Safety Alignment [23.776344382492997]
  We show that aligned LLMs encode harmfulness and refusal as separable directions in the residual stream at prompt-side token positions.<n>We introduce HARC, a fine-tuning method that pairs the two directions across both prompt and response positions.
  arXiv  Detail & Related papers  (2026-07-01T07:58:16Z)
• Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories [9.386028796990399]
  Safety-aligned Large Language Models (LLMs) remain vulnerable to interventions during inference that redirect generation toward harmful outputs.<n>Recent work attributes this to shallow safety, where alignment concentrates in the first few output tokens.<n>We show that short token injections at any generation step can substantially alter subsequent safety behavior.
  arXiv  Detail & Related papers  (2026-06-03T12:01:27Z)
• THINKSAFE: Self-Generated Safety Alignment for Reasoning Models [60.10077024249373]
  We propose ThinkSafe, a framework that restores safety alignment without external teachers.<n>Our key insight is that while compliance suppresses safety mechanisms, models often retain latent knowledge to identify harm.<n> Experiments on DeepSeek-R1-Distill and Qwen3 show ThinkSafe significantly improves safety while preserving reasoning proficiency.
  arXiv  Detail & Related papers  (2026-01-30T16:31:02Z)
• Understanding and Preserving Safety in Fine-Tuned LLMs [20.821783178639063]
  Fine-tuning can substantially degrade safety alignment, even when the fine-tuning data is harmless.<n>We propose safety-preserving fine-tuning (SPF), a lightweight approach that explicitly removes gradient components conflicting with the low-rank safety subspace.<n> SPF consistently maintains downstream task performance and recovers nearly all pre-trained safety alignment, even under adversarial fine-tuning scenarios.
  arXiv  Detail & Related papers  (2026-01-15T07:33:13Z)
• When Should We Introduce Safety Interventions During Pretraining? [100.3502954292386]
  Prior work has shown that interventions applied during pretraining, such as rephrasing harmful content, can substantially improve the safety of the resulting models.<n>We find that introducing interventions earlier generally yields more robust models with no increase in overrefusal rates.<n>We also see clear benefits in the steerability of models towards safer generations.
  arXiv  Detail & Related papers  (2026-01-11T22:38:17Z)
• Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Models [93.5740266114488]
  Constructive Safety Alignment (CSA) protects against malicious misuse while actively guiding vulnerable users toward safe and helpful results.<n>Oy1 achieves state-of-the-art safety among open models while retaining high general capabilities.<n>We release Oy1, code, and the benchmark to support responsible, user-centered AI.
  arXiv  Detail & Related papers  (2025-09-02T03:04:27Z)
• Probing the Robustness of Large Language Models Safety to Latent Perturbations [30.16804362984161]
  Safety alignment is a key requirement for building reliable Artificial General Intelligence.<n>We observe that minor latent shifts can still trigger unsafe responses in aligned models.<n>We introduce Layer-wise Adversarial Patch Training(LAPT), a fine-tuning strategy that injects controlled perturbations into hidden representations during training.
  arXiv  Detail & Related papers  (2025-06-19T07:03:05Z)
• Shape it Up! Restoring LLM Safety during Finetuning [65.75757313781104]
  Finetuning large language models (LLMs) enables user-specific customization but introduces critical safety risks.<n>We propose dynamic safety shaping (DSS), a framework that uses fine-grained safety signals to reinforce learning from safe segments of a response while suppressing unsafe content.<n>We present STAR-DSS, guided by STAR scores, that robustly mitigates finetuning risks and delivers substantial safety improvements across diverse threats, datasets, and model families.
  arXiv  Detail & Related papers  (2025-05-22T18:05:16Z)
• Improving LLM Safety Alignment with Dual-Objective Optimization [81.98466438000086]
  Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks.<n>We propose an improved safety alignment that disentangles DPO objectives into two components: (1) robust refusal training, which encourages refusal even when partial unsafe generations are produced, and (2) targeted unlearning of harmful knowledge.
  arXiv  Detail & Related papers  (2025-03-05T18:01:05Z)
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep [48.823599143711235]
  The safety alignment of current Large Language Models (LLMs) is vulnerable. Relatively simple attacks, or even benign fine-tuning, can jailbreak aligned models. We show that deepening the safety alignment beyond just the first few tokens can often meaningfully improve robustness against some common exploits.
  arXiv  Detail & Related papers  (2024-06-10T00:35:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.