Black-box Optimization of LLM Outputs by Asking for Directions

• URL: http://arxiv.org/abs/2510.16794v1
• Date: Sun, 19 Oct 2025 11:13:45 GMT
• Title: Black-box Optimization of LLM Outputs by Asking for Directions
• Authors: Jie Zhang, Meng Ding, Yang Liu, Jue Hong, Florian Tramèr,
• Abstract summary: We present a novel approach for attacking black-box large language models (LLMs) by exploiting their ability to express confidence in natural language.<n>We apply our general method to three attack scenarios: adversarial examples for vision-LLMs, jailbreaks and prompt injections.
• Score: 34.0051902705951
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a novel approach for attacking black-box large language models (LLMs) by exploiting their ability to express confidence in natural language. Existing black-box attacks require either access to continuous model outputs like logits or confidence scores (which are rarely available in practice), or rely on proxy signals from other models. Instead, we demonstrate how to prompt LLMs to express their internal confidence in a way that is sufficiently calibrated to enable effective adversarial optimization. We apply our general method to three attack scenarios: adversarial examples for vision-LLMs, jailbreaks and prompt injections. Our attacks successfully generate malicious inputs against systems that only expose textual outputs, thereby dramatically expanding the attack surface for deployed LLMs. We further find that better and larger models exhibit superior calibration when expressing confidence, creating a concerning security paradox where model capability improvements directly enhance vulnerability. Our code is available at this [link](https://github.com/zj-jayzhang/black_box_llm_optimization).

Related papers

• RL-MTJail: Reinforcement Learning for Automated Black-Box Multi-Turn Jailbreaking of Large Language Models [60.201244463046784]
  Large language models are vulnerable to jailbreak attacks.<n>This paper studies black-box multi-turn jailbreaks, aiming to train attacker LLMs to elicit harmful content from black-box models.
  arXiv  Detail & Related papers  (2025-12-08T17:42:59Z)
• PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization [0.0]
  This paper introduces a novel framework for hardening system prompts through shield appending.<n>We leverage an LLM-as-optimizer to search the space of possible SHIELDs, seeking to minimize a leakage metric derived from a suite of adversarial attacks.<n>We demonstrate empirically that our optimized SHIELDs significantly reduce prompt leakage against a comprehensive set of extraction attacks.
  arXiv  Detail & Related papers  (2025-11-20T10:25:45Z)
• Friend or Foe: How LLMs' Safety Mind Gets Fooled by Intent Shift Attack [53.34204977366491]
  Large language models (LLMs) remain vulnerable to jailbreaking attacks despite their impressive capabilities.<n>In this paper, we introduce ISA (Intent Shift Attack), which obfuscates LLMs about the intent of the attacks.<n>Our approach only needs minimal edits to the original request, and yields natural, human-readable, and seemingly harmless prompts.
  arXiv  Detail & Related papers  (2025-11-01T13:44:42Z)
• Context Misleads LLMs: The Role of Context Filtering in Maintaining Safe Alignment of LLMs [7.183743839410718]
  Malicious users often exploit adversarial context to deceive Large Language Models (LLMs)<n>In this study, we propose a new defense mechanism called Context Filtering model.<n>Our model demonstrates its ability to reduce the Attack Success Rates of jailbreak attacks by up to 88%.
  arXiv  Detail & Related papers  (2025-08-09T02:37:59Z)
• Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models [20.99874786089634]
  Previous jailbreak attacks often inject malicious instructions from text into less aligned modalities, such as vision.<n>We propose a novel implicit jailbreak framework termed IJA that stealthily embeds malicious instructions into images via at least significant bit steganography.<n>On commercial models like GPT-4o and Gemini-1.5 Pro, our method achieves attack success rates of over 90% using an average of only 3 queries.
  arXiv  Detail & Related papers  (2025-05-22T09:34:47Z)
• `Do as I say not as I do': A Semi-Automated Approach for Jailbreak Prompt Attack against Multimodal LLMs [33.49407213040455]
  We introduce the first voice-based jailbreak attack against multimodal large language models (LLMs)<n>We propose a novel strategy, in which the disallowed prompt is flanked by benign, narrative-driven prompts.<n>We demonstrate that Flanking Attack is capable of manipulating state-of-the-art LLMs into generating misaligned and forbidden outputs.
  arXiv  Detail & Related papers  (2025-02-02T10:05:08Z)
• An Interpretable N-gram Perplexity Threat Model for Large Language Model Jailbreaks [87.64278063236847]
  In this work, we propose a unified threat model for the principled comparison of jailbreak attacks.<n>Our threat model checks if a given jailbreak is likely to occur in the distribution of text.<n>We adapt popular attacks to this threat model, and, for the first time, benchmark these attacks on equal footing with it.
  arXiv  Detail & Related papers  (2024-10-21T17:27:01Z)
• AdaPPA: Adaptive Position Pre-Fill Jailbreak Attack Approach Targeting LLMs [34.221522224051846]
  We propose an adaptive position pre-fill jailbreak attack approach for executing jailbreak attacks on Large Language Models (LLMs) Our method leverages the model's instruction-following capabilities to first output safe content, then exploits its narrative-shifting abilities to generate harmful content. Our method can improve the attack success rate by 47% on the widely recognized secure model (Llama2) compared to existing approaches.
  arXiv  Detail & Related papers  (2024-09-11T00:00:58Z)
• Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models? [3.258629327038072]
  Large Language Models (LLMs) have demonstrated impressive capabilities in natural language tasks. Yet, the potential for generating harmful content through these models seems to persist. This paper explores the concept of jailbreaking LLMs-reversing their alignment through adversarial triggers.
  arXiv  Detail & Related papers  (2024-08-05T17:27:29Z)
• AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting [54.931241667414184]
  We propose textbfAdaptive textbfShield Prompting, which prepends inputs with defense prompts to defend MLLMs against structure-based jailbreak attacks. Our methods can consistently improve MLLMs' robustness against structure-based jailbreak attacks.
  arXiv  Detail & Related papers  (2024-03-14T15:57:13Z)
• Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models [79.0183835295533]
  We introduce the first benchmark for indirect prompt injection attacks, named BIPIA, to assess the risk of such vulnerabilities.<n>Our analysis identifies two key factors contributing to their success: LLMs' inability to distinguish between informational context and actionable instructions, and their lack of awareness in avoiding the execution of instructions within external content.<n>We propose two novel defense mechanisms-boundary awareness and explicit reminder-to address these vulnerabilities in both black-box and white-box settings.
  arXiv  Detail & Related papers  (2023-12-21T01:08:39Z)
• Universal and Transferable Adversarial Attacks on Aligned Language Models [118.41733208825278]
  We propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors. Surprisingly, we find that the adversarial prompts generated by our approach are quite transferable.
  arXiv  Detail & Related papers  (2023-07-27T17:49:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.