Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking

• URL: http://arxiv.org/abs/2605.08778v1
• Date: Sat, 09 May 2026 08:07:30 GMT
• Title: Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreaking
• Abstract summary: We propose TRACE, a turn-aware credit assignment framework for reinforcement learning-based multi-turn jailbreaking.<n>For successful trajectories, TRACE estimates turn-level contributions via leave-one-turn-out semantic masking.<n>For failed ones, TRACE assigns penalties based on prompt harmfulness and semantic relevance.
• Score: 39.11709072420052
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deploying LLMs in multi-turn dialogues facilitates jailbreak attacks that distribute harmful intent across seemingly benign turns. Recent training-based multi-turn jailbreak methods learn long-horizon attack strategies from interaction feedback, but often rely on coarse trajectory-level outcome signals that broadcast uniformly to every turn. However, we find that turn-level contributions in multi-turn jailbreaking are non-uniform, phase-dependent, and target-specific. Such coarse outcome supervision induces a credit assignment problem, leading to over-rewarding redundant turns in successful trajectories and under-crediting useful intermediate turns in failed ones. To address this, we propose TRACE, a turn-aware credit assignment framework for reinforcement learning (RL)-based multi-turn jailbreaking. For successful trajectories, TRACE estimates turn-level contributions via leave-one-turn-out semantic masking; for failed ones, TRACE assigns penalties based on prompt harmfulness and semantic relevance, with an additional local refusal-aware penalty. Furthermore, we reuse the attack-side credit signal for multi-turn defense alignment. Extensive experiments on open-source and closed-source targets show that TRACE achieves strong overall performance in effectiveness, transferability, and efficiency, yielding about a 25% relative improvement in attack success rate over the strongest RL baseline while also improving the safety-utility balance when reused for defense alignment.

Related papers

• HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation [50.53459634301361]
  HERO is a hindsight-enhanced self-distillation framework that uses next environment observations as locally aligned feedback.<n> HERO improves task success and reduces unnecessary turns over environment-feedback-only self-distillation and GRPO.
  arXiv  Detail & Related papers  (2026-06-10T01:35:34Z)
• HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents [45.00250014235802]
  Training long-horizon LLM agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected.<n>Recent methods alleviate this issue by generating rewards or textual hints from turn-level action-output signals, or by using feedback-conditioned self-distillation.<n>We propose HINT-SD, a targeted self-distillation framework that uses full-trajectory hindsight to select failure-relevant actions and applies feedback-conditioned distillation only on targeted action spans.
  arXiv  Detail & Related papers  (2026-05-18T05:34:03Z)
• SEMA: Simple yet Effective Learning for Multi-Turn Jailbreak Attacks [53.97948802255959]
  We propose a framework that trains a multi-turn attacker without relying on any existing strategies or external data.<n>Prefilling self-tuning enables usable rollouts by fine-tuning on non-refusal, well-structured, multi-turn adversarial prompts.<n>We anchor harmful intent in multi-turn jailbreaks via an intent-drift-aware reward that combines intent alignment, compliance risk, and level of detail.
  arXiv  Detail & Related papers  (2026-02-06T16:44:57Z)
• R$^3$L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification [44.99719889905381]
  Reinforcement learning drives recent advances in reasoning and agentic capabilities, yet current approaches struggle with both exploration and exploitation.<n>We propose R$3$L, Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification.<n>Experiments on agentic and reasoning tasks demonstrate 5% to 52% relative improvements over baselines while maintaining training stability.
  arXiv  Detail & Related papers  (2026-01-07T09:04:52Z)
• RL-MTJail: Reinforcement Learning for Automated Black-Box Multi-Turn Jailbreaking of Large Language Models [60.201244463046784]
  Large language models are vulnerable to jailbreak attacks.<n>This paper studies black-box multi-turn jailbreaks, aiming to train attacker LLMs to elicit harmful content from black-box models.
  arXiv  Detail & Related papers  (2025-12-08T17:42:59Z)
• bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs [33.470999703070866]
  Existing approaches to embedding jailbreak triggers suffer from limitations including poor generalization, compromised stealthiness, or reduced contextual usability.<n>We propose bi-GRPO, a novel RL-based framework tailored explicitly for jailbreak backdoor injection.
  arXiv  Detail & Related papers  (2025-09-24T05:56:41Z)
• A Representation Engineering Perspective on the Effectiveness of Multi-Turn Jailbreaks [3.8246557700763715]
  We study the effectiveness of the Crescendo multi-turn jailbreak at the level of intermediate model representations.<n>Our results help explain why single-turn jailbreak defenses are generally ineffective against multi-turn attacks.
  arXiv  Detail & Related papers  (2025-06-29T23:28:55Z)
• GRAF: Multi-turn Jailbreaking via Global Refinement and Active Fabrication [55.63412213263305]
  Large Language Models pose notable safety risks due to potential misuse for malicious purposes.<n>We propose a novel multi-turn jailbreaking method that globally refines the attack trajectory at each interaction.<n>In addition, we actively fabricate model responses to suppress safety-related warnings, thereby increasing the likelihood of eliciting harmful outputs.
  arXiv  Detail & Related papers  (2025-06-22T03:15:05Z)
• Improving LLM Safety Alignment with Dual-Objective Optimization [81.98466438000086]
  Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks.<n>We propose an improved safety alignment that disentangles DPO objectives into two components: (1) robust refusal training, which encourages refusal even when partial unsafe generations are produced, and (2) targeted unlearning of harmful knowledge.
  arXiv  Detail & Related papers  (2025-03-05T18:01:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.