On the Geometric Limits of Transformer Defenses against Obfuscation Attacks: Latent Embedding Collapse & Performance Robustness Gap

• URL: http://arxiv.org/abs/2605.19159v1
• Date: Mon, 18 May 2026 22:25:47 GMT
• Title: On the Geometric Limits of Transformer Defenses against Obfuscation Attacks: Latent Embedding Collapse & Performance Robustness Gap
• Abstract summary: We show that high detection performance does not imply representational robustness.<n>Results indicate that across multiple BERT family encoders with varying depth and capacity, detectors achieve near-perfect classification performance.<n>Obfuscated embeddings further exhibit elevated intra-class variance (3.33 +/- 6.23), indicating severe latent-space instability despite high performance.
• Score: 0.6445605125467572
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt injection attacks pose significant risks to language model safety, yet existing defenses are typically evaluated using classification performance. We show that high detection performance does not imply representational robustness. Specifically, multi-operator obfuscated prompts (combining homoglyphs, zero-width characters, and punctuation or emoji noise) can partially collapse onto the embedding manifold of clean prompts, a phenomenon we term latent embedding collapse. Results indicate that across multiple BERT family encoders with varying depth and capacity, detectors achieve near-perfect classification performance, yet the minimal clean-obfuscated margin delta = 1.02, indicating near-overlap of obfuscated and clean embeddings. Obfuscated embeddings further exhibit elevated intra-class variance (3.33 +/- 6.23), indicating severe latent-space instability despite high performance. These results reveal a substantial perf ormance-robustness gap, demonstrating that standard evaluation metrics fail to capture latent embedding collapse and underlying geometric fragility. Our findings show that increasing model capacity does not eliminate latent embedding collapse, motivating geometry-aware robustness analysis as a necessary complement to performance-based evaluation for prompt-injection defenses.

Related papers

• Geometric Collapse: When Vision Models Fail to Verify Physical Causality [52.43819179934414]
  We propose Scrambled Edges, a controlled counterfactual that injects salient edge-like cues.<n>With energy-matched and structure-matched controls, we isolate the effect of unsupported edge evidence from high-frequency energy and edge sparsity.<n>Across CNN/ViT/SSL depth predictors, Scrambled Edges induce up to 3.2x larger deviation from clean predictions than energy-matched noise.
  arXiv  Detail & Related papers  (2026-07-08T00:04:02Z)
• The Illusion of High Utility in Safety Alignment of Text-to-Image Diffusion Models [49.67749928542536]
  Safety alignment of text-to-image (T2I) diffusion models aims to suppress harmful generations while preserving utility on benign prompts.<n>Recent methods often appear to deliver high safety with high utility, but this conclusion rests largely on coarse global utility metrics.<n>We show that when utility is measured with structured evaluation, this illusion breaks: onA (Text-to-Image Faithfulness evaluation with Question Answering)<n>We propose StructureAware Geometric Regularization (SAGE), a safety alignment objective that explicitly preserves embedding spread and inter-prompt relational structure.
  arXiv  Detail & Related papers  (2026-07-01T04:00:27Z)
• Beyond Red-Teaming: Formal Guarantees of LLM Guardrail Classifiers [1.9839136494100942]
  Guardrails defend production language models against harmful behavior, but they provide no formal guarantees.<n>We propose two constructions of such regions: SVD-aligned hyper-rectangles and Gaussian Mixture Models.<n>Applying this framework to three author-trained Guardrails on the toxicity domain, every hyper-rectangle configuration returns SAT.<n>While GPT-2 and Llama-3.1-8B maintain robust coverage of 90% and 80% across varying boundaries, BERT's safety guarantees prove uniquely volatile.
  arXiv  Detail & Related papers  (2026-05-11T17:41:38Z)
• TwinGate: Stateful Defense against Decompositional Jailbreaks in Untraceable Traffic via Asymmetric Contrastive Learning [60.68349524623048]
  Decompositional jailbreaks pose a critical threat to large language models.<n>We present TwinGate, a stateful dual-encoder defense framework.<n>We construct a comprehensive dataset of over 3.62 million instructions spanning 8,600 distinct malicious intents.
  arXiv  Detail & Related papers  (2026-04-30T13:44:01Z)
• Splats in Splats++: Robust and Generalizable 3D Gaussian Splatting Steganography [32.33494908495543]
  3D Gaussian Splatting (3DGS) has recently redefined the paradigm of 3D reconstruction, striking an unprecedented balance between visual fidelity and computational efficiency.<n>We present textbftextitSplats in Splats++, a unified and pipeline-agnostic steganography framework that seamlessly embeds high-capacity 3D/4D content directly within the native 3DGS representation.
  arXiv  Detail & Related papers  (2026-04-17T09:13:20Z)
• DeepSeek Robustness Against Semantic-Character Dual-Space Mutated Prompt Injection [45.67420390185547]
  We propose PromptFuzz-SC, a semantic-character dual-space mutation framework for evaluating robustness against prompt injection.<n>A hybrid search strategy combining epsilon-greedy exploration and hill-climbing refinement is adopted to efficiently discover high-quality adversarial prompts.<n> Experimental results on DeepSeek demonstrate that dual-space mutation achieves the strongest overall attack performance.
  arXiv  Detail & Related papers  (2026-04-14T10:20:15Z)
• BadDet+: Robust Backdoor Attacks for Object Detection [10.393154496941527]
  We introduce BadDet+, a penalty-based framework that unifies Region Misclassification Attacks (RMA) and Object Disappearance Attacks (ODA)<n>On real-world benchmarks, BadDet+ achieves superior synthetic-to-physical transfer compared to existing RMA and ODA baselines while preserving clean performance.<n>These results highlight significant vulnerabilities in object detection and the necessity for specialized defenses.
  arXiv  Detail & Related papers  (2026-01-28T21:46:33Z)
• Exploring Semantic-constrained Adversarial Example with Instruction Uncertainty Reduction [51.50282796099369]
  This paper develops a multi-dimensional instruction uncertainty reduction framework to generate semantically constrained adversarial examples.<n>By predicting the language-guided sampling process, the optimization process will be stabilized by the designed ResAdv-DDIM sampler.<n>We realize the reference-free generation of semantically constrained 3D adversarial examples for the first time.
  arXiv  Detail & Related papers  (2025-10-27T04:02:52Z)
• ForensicsSAM: Toward Robust and Unified Image Forgery Detection and Localization Resisting to Adversarial Attack [56.0056378072843]
  We show that highly transferable adversarial images can be crafted solely via the upstream model.<n>We propose ForensicsSAM, a unified IFDL framework with built-in adversarial robustness.
  arXiv  Detail & Related papers  (2025-08-10T16:03:44Z)
• CeTAD: Towards Certified Toxicity-Aware Distance in Vision Language Models [16.5022773312661]
  We propose a universal certified defence framework to safeguard large vision-language models against jailbreak attacks.<n>First, we proposed a novel distance metric to quantify semantic discrepancies between malicious and intended responses.<n>Then, we devise a regressed certification approach that employs randomized smoothing to provide formal robustness guarantees.
  arXiv  Detail & Related papers  (2025-03-08T17:33:55Z)
• Clustering Effect of (Linearized) Adversarial Robust Models [60.25668525218051]
  We propose a novel understanding of adversarial robustness and apply it on more tasks including domain adaption and robustness boosting. Experimental evaluations demonstrate the rationality and superiority of our proposed clustering strategy.
  arXiv  Detail & Related papers  (2021-11-25T05:51:03Z)
• Exploring Robustness of Unsupervised Domain Adaptation in Semantic Segmentation [74.05906222376608]
  We propose adversarial self-supervision UDA (or ASSUDA) that maximizes the agreement between clean images and their adversarial examples by a contrastive loss in the output space. This paper is rooted in two observations: (i) the robustness of UDA methods in semantic segmentation remains unexplored, which pose a security concern in this field; and (ii) although commonly used self-supervision (e.g., rotation and jigsaw) benefits image tasks such as classification and recognition, they fail to provide the critical supervision signals that could learn discriminative representation for segmentation tasks.
  arXiv  Detail & Related papers  (2021-05-23T01:50:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.