Geometry-Guided Adversarial Prompt Detection via Curvature and Local Intrinsic Dimension

• URL: http://arxiv.org/abs/2503.03502v2
• Date: Tue, 07 Oct 2025 10:03:12 GMT
• Title: Geometry-Guided Adversarial Prompt Detection via Curvature and Local Intrinsic Dimension
• Authors: Canaan Yung, Hanxun Huang, Christopher Leckie, Sarah Erfani,
• Abstract summary: CurvaLID is a novel defence framework that efficiently detects adversarial prompts by leveraging their geometric properties.<n>CurvaLID builds on the geometric analysis of text prompts to uncover their underlying differences.<n>Our findings show that adversarial prompts exhibit distinct geometric signatures from benign prompts, enabling CurvaLID to achieve near-perfect classification.
• Score: 10.892846618107392
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adversarial prompts are capable of jailbreaking frontier large language models (LLMs) and inducing undesirable behaviours, posing a significant obstacle to their safe deployment. Current mitigation strategies primarily rely on activating built-in defence mechanisms or fine-tuning LLMs, both of which are computationally expensive and can sacrifice model utility. In contrast, detection-based approaches are more efficient and practical for deployment in real-world applications. However, the fundamental distinctions between adversarial and benign prompts remain poorly understood. In this work, we introduce CurvaLID, a novel defence framework that efficiently detects adversarial prompts by leveraging their geometric properties. It is agnostic to the type of LLM, offering a unified detection framework across diverse adversarial prompts and LLM architectures. CurvaLID builds on the geometric analysis of text prompts to uncover their underlying differences. We theoretically extend the concept of curvature via the Whewell equation into an $n$-dimensional word embedding space, enabling us to quantify local geometric properties, including semantic shifts and curvature in the underlying manifolds. To further enhance our solution, we leverage Local Intrinsic Dimensionality (LID) to capture complementary geometric features of text prompts within adversarial subspaces. Our findings show that adversarial prompts exhibit distinct geometric signatures from benign prompts, enabling CurvaLID to achieve near-perfect classification and outperform state-of-the-art detectors in adversarial prompt detection. CurvaLID provides a reliable and efficient safeguard against malicious queries as a model-agnostic method that generalises across multiple LLMs and attack families.

Related papers

• Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs [85.69785384599827]
  Human-object interaction (HOI) detection aims to localize human-object pairs and the interactions between them.<n>Existing methods operate under a closed-world assumption, treating the task as a classification problem over a small, predefined verb set.<n>We propose GRASP-HO, a novel Generative Reasoning And Steerable Perception framework that reformulates HOI detection from the closed-set classification task to the open-vocabulary generation problem.
  arXiv  Detail & Related papers  (2025-12-19T14:41:50Z)
• Geometry-Aware Backdoor Attacks: Leveraging Curvature in Hyperbolic Embeddings [3.8806403512213787]
  Non-Euclidean foundation models place representations in curved spaces such as hyperbolic geometry.<n>Small input changes appear subtle to standard input-space detectors but produce disproportionately large shifts in the model's representation space.<n>We propose a geometry-adaptive trigger and evaluate it across tasks and architectures.
  arXiv  Detail & Related papers  (2025-10-07T19:24:43Z)
• Are All Prompt Components Value-Neutral? Understanding the Heterogeneous Adversarial Robustness of Dissected Prompt in Large Language Models [11.625319498017733]
  We introduce PromptAnatomy, an automated framework that dissects prompts into functional components.<n>We generate adversarial examples by selectively perturbing each component using our proposed method, ComPerturb.<n>As a complementary resource, we annotate four public instruction-tuning datasets using the PromptAnatomy framework.
  arXiv  Detail & Related papers  (2025-08-03T02:46:30Z)
• Large Language Models Encode Semantics in Low-Dimensional Linear Subspaces [31.401762286885656]
  Understanding the space geometry of large language models (LLMs) is key to interpreting their behavior and improving alignment.<n>baturayWe investigate what extent LLMs internally organize related to semantic understanding.
  arXiv  Detail & Related papers  (2025-07-13T17:03:25Z)
• Curved Inference: Concern-Sensitive Geometry in Large Language Model Residual Streams [0.0]
  We propose a geometric Interpretability framework that tracks how the residual stream trajectory of a large language model bends in response to shifts in semantic concern.<n>We analyse Gemma3-1b and LLaMA3.2-3b using five native-space metrics, with a primary focus on curvature (kappa_i) and salience (S(t))<n>We find that concern-shifted prompts reliably alter internal activation trajectories in both models.
  arXiv  Detail & Related papers  (2025-07-08T23:05:00Z)
• AdversariaL attacK sAfety aLIgnment(ALKALI): Safeguarding LLMs through GRACE: Geometric Representation-Aware Contrastive Enhancement- Introducing Adversarial Vulnerability Quality Index (AVQI) [21.209413521884297]
  Adversarial threats against LLMs are escalating faster than current defenses can adapt.<n>We introduce ALKALI, the first rigorously curated adversarial benchmark.<n>We introduce GRACE, an alignment framework coupling preference learning with latent space regularization.
  arXiv  Detail & Related papers  (2025-06-10T15:14:17Z)
• A Few Large Shifts: Layer-Inconsistency Based Minimal Overhead Adversarial Example Detection [13.109309606764754]
  We introduce a plug-in detection framework that leverages internal layer-wise inconsistencies within the target model itself.<n>Our method achieves state-of-the-art detection performance with negligible computational overhead.
  arXiv  Detail & Related papers  (2025-05-19T00:48:53Z)
• Towards Copyright Protection for Knowledge Bases of Retrieval-augmented Language Models via Reasoning [58.57194301645823]
  Large language models (LLMs) are increasingly integrated into real-world personalized applications.<n>The valuable and often proprietary nature of the knowledge bases used in RAG introduces the risk of unauthorized usage by adversaries.<n>Existing methods that can be generalized as watermarking techniques to protect these knowledge bases typically involve poisoning or backdoor attacks.<n>We propose name for harmless' copyright protection of knowledge bases.
  arXiv  Detail & Related papers  (2025-02-10T09:15:56Z)
• Compromising Embodied Agents with Contextual Backdoor Attacks [69.71630408822767]
  Large language models (LLMs) have transformed the development of embodied intelligence. This paper uncovers a significant backdoor security threat within this process. By poisoning just a few contextual demonstrations, attackers can covertly compromise the contextual environment of a black-box LLM.
  arXiv  Detail & Related papers  (2024-08-06T01:20:12Z)
• Prompt Leakage effect and defense strategies for multi-turn LLM interactions [95.33778028192593]
  Leakage of system prompts may compromise intellectual property and act as adversarial reconnaissance for an attacker. We design a unique threat model which leverages the LLM sycophancy effect and elevates the average attack success rate (ASR) from 17.7% to 86.2% in a multi-turn setting. We measure the mitigation effect of 7 black-box defense strategies, along with finetuning an open-source model to defend against leakage attempts.
  arXiv  Detail & Related papers  (2024-04-24T23:39:58Z)
• Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs [9.624124576891075]
  Existing alignment methods can lead large language models (LLMs) to be Adaptive Chameleons when external evidence conflicts with their parametric memory. We propose a novel framework: Dialectical Alignment (DA), which utilizes AI feedback to identify optimal strategies for LLMs to navigate inter-context conflicts. Our experiments show that the DA improves poisoned data attack defense by 20 and does not require any additional prompt engineering.
  arXiv  Detail & Related papers  (2024-03-30T22:41:05Z)
• NLP Verification: Towards a General Methodology for Certifying Robustness [9.897538432223714]
  Machine Learning (ML) has exhibited substantial success in the field of Natural Language Processing (NLP)<n>As these systems are increasingly integrated into real-world applications, ensuring their safety and reliability becomes a primary concern.<n>We propose a general methodology to analyse the effect of the embedding gap, a problem that refers to the discrepancy between verification of geometric subspaces and the semantic meaning of sentences.
  arXiv  Detail & Related papers  (2024-03-15T09:43:52Z)
• Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension [63.330262740414646]
  We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs) We suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations.
  arXiv  Detail & Related papers  (2024-02-28T04:56:21Z)
• ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings [58.82536530615557]
  We propose an Adversarial Suffix Embedding Translation Framework (ASETF) to transform continuous adversarial suffix embeddings into coherent and understandable text. Our method significantly reduces the computation time of adversarial suffixes and achieves a much better attack success rate to existing techniques.
  arXiv  Detail & Related papers  (2024-02-25T06:46:27Z)
• A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia [57.31074448586854]
  Large language models (LLMs) have an impressive ability to draw on novel information supplied in their context. Yet the mechanisms underlying this contextual grounding remain unknown. We present a novel method to study grounding abilities using Fakepedia.
  arXiv  Detail & Related papers  (2023-12-04T17:35:42Z)
• Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generation [15.77263269398368]
  Large Language Models (LLMs) drive current AI breakthroughs. We shed the light on LLMs inner mechanisms through the lens of geometry. We derive interpretable geometrical features that can be extracted from any (pre-trained) LLM.
  arXiv  Detail & Related papers  (2023-12-04T06:01:32Z)
• Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information [67.78183175605761]
  Large Language Models are susceptible to adversarial prompt attacks. This vulnerability underscores a significant concern regarding the robustness and reliability of LLMs. We introduce a novel approach to detecting adversarial prompts at a token level.
  arXiv  Detail & Related papers  (2023-11-20T03:17:21Z)
• Transcending Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection [57.646582245834324]
  We propose a simple yet effective deepfake detector called LSDA. It is based on a idea: representations with a wider variety of forgeries should be able to learn a more generalizable decision boundary. We show that our proposed method is surprisingly effective and transcends state-of-the-art detectors across several widely used benchmarks.
  arXiv  Detail & Related papers  (2023-11-19T09:41:10Z)
• Red Teaming Language Model Detectors with Language Models [114.36392560711022]
  Large language models (LLMs) present significant safety and ethical risks if exploited by malicious users. Recent works have proposed algorithms to detect LLM-generated text and protect LLMs. We study two types of attack strategies: 1) replacing certain words in an LLM's output with their synonyms given the context; 2) automatically searching for an instructional prompt to alter the writing style of the generation.
  arXiv  Detail & Related papers  (2023-05-31T10:08:37Z)
• GELATO: Geometrically Enriched Latent Model for Offline Reinforcement Learning [54.291331971813364]
  offline reinforcement learning approaches can be divided into proximal and uncertainty-aware methods. In this work, we demonstrate the benefit of combining the two in a latent variational model. Our proposed metrics measure both the quality of out of distribution samples as well as the discrepancy of examples in the data.
  arXiv  Detail & Related papers  (2021-02-22T19:42:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.