Adversarial Lens: Exploiting Attention Layers to Generate Adversarial Examples for Evaluation

• URL: http://arxiv.org/abs/2512.23837v1
• Date: Mon, 29 Dec 2025 19:59:52 GMT
• Title: Adversarial Lens: Exploiting Attention Layers to Generate Adversarial Examples for Evaluation
• Abstract summary: We generate adversarial examples directly from attention-layer token distributions.<n>We evaluate whether tokens extracted from intermediate layers can serve as effective adversarial perturbations for downstream evaluation tasks.
• Score: 0.2800045883776013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in mechanistic interpretability suggest that intermediate attention layers encode token-level hypotheses that are iteratively refined toward the final output. In this work, we exploit this property to generate adversarial examples directly from attention-layer token distributions. Unlike prompt-based or gradient-based attacks, our approach leverages model-internal token predictions, producing perturbations that are both plausible and internally consistent with the model's own generation process. We evaluate whether tokens extracted from intermediate layers can serve as effective adversarial perturbations for downstream evaluation tasks. We conduct experiments on argument quality assessment using the ArgQuality dataset, with LLaMA-3.1-Instruct-8B serving as both the generator and evaluator. Our results show that attention-based adversarial examples lead to measurable drops in evaluation performance while remaining semantically similar to the original inputs. However, we also observe that substitutions drawn from certain layers and token positions can introduce grammatical degradation, limiting their practical effectiveness. Overall, our findings highlight both the promise and current limitations of using intermediate-layer representations as a principled source of adversarial examples for stress-testing LLM-based evaluation pipelines.

Related papers

• When Benchmarks Leak: Inference-Time Decontamination for LLMs [4.071875179293035]
  DeconIEP operates entirely during evaluation by applying small, bounded perturbations in the input embedding space.<n>We propose DeconIEP, a decontamination framework that operates entirely during evaluation by applying small, bounded perturbations in the input embedding space.
  arXiv  Detail & Related papers  (2026-01-27T08:19:40Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• Context-Adaptive Requirements Defect Prediction through Human-LLM Collaboration [1.4499356176178066]
  We propose a Human-LLM Collaboration (HLC) approach that treats defect prediction as an adaptive process rather than a static classification task.<n>We evaluate this approach using the weak word smell on the QuRE benchmark of 1,266 annotated Mercedes-Benz requirements.
  arXiv  Detail & Related papers  (2026-01-05T10:00:14Z)
• Aligning the Evaluation of Probabilistic Predictions with Downstream Value [2.6636053598505307]
  Metrics based solely on predictive performance often diverge from measures of real-world downstream impact.<n>We propose a data-driven method to learn a proxy evaluation function aligned with the downstream evaluation.<n>Our approach leverages weighted scoring rules parametrized by a neural network, where weighting is learned to align with the performance in the downstream task.
  arXiv  Detail & Related papers  (2025-08-25T17:41:27Z)
• ScoreAdv: Score-based Targeted Generation of Natural Adversarial Examples via Diffusion Models [13.250999667915254]
  In this paper, we introduce a novel approach for generating adversarial examples based on diffusion models, named ScoreAdv.<n>Our method is capable of generating an unlimited number of natural adversarial examples and can attack not only classification models but also retrieval models.<n>Our results demonstrate that ScoreAdv achieves state-of-the-art attack success rates and image quality, while maintaining inference efficiency.
  arXiv  Detail & Related papers  (2025-07-08T15:17:24Z)
• Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability [53.51560766150442]
  Critical tokens are elements within reasoning trajectories that significantly influence incorrect outcomes.<n>We present a novel framework for identifying these tokens through rollout sampling.<n>We show that identifying and replacing critical tokens significantly improves model accuracy.
  arXiv  Detail & Related papers  (2024-11-29T18:58:22Z)
• AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation [57.8363998797433]
  We propose AMRFact, a framework that generates perturbed summaries using Abstract Meaning Representations (AMRs) Our approach parses factually consistent summaries into AMR graphs and injects controlled factual inconsistencies to create negative examples, allowing for coherent factually inconsistent summaries to be generated with high error-type coverage.
  arXiv  Detail & Related papers  (2023-11-16T02:56:29Z)
• Time-series Generation by Contrastive Imitation [87.51882102248395]
  We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
  arXiv  Detail & Related papers  (2023-11-02T16:45:25Z)
• Outcome-Guided Counterfactuals for Reinforcement Learning Agents from a Jointly Trained Generative Latent Space [0.0]
  We present a novel generative method for producing unseen and plausible counterfactual examples for reinforcement learning (RL) agents. Our approach uses a variational autoencoder to train a latent space that jointly encodes information about the observations and outcome variables pertaining to an agent's behavior.
  arXiv  Detail & Related papers  (2022-07-15T19:09:54Z)
• Unreasonable Effectiveness of Last Hidden Layer Activations [0.5156484100374058]
  We show that using some widely known activation functions in the output layer of the model with high temperature values has the effect of zeroing out the gradients for both targeted and untargeted attack cases. We've experimentally verified the efficacy of our approach on MNIST (Digit), CIFAR10 datasets.
  arXiv  Detail & Related papers  (2022-02-15T12:02:59Z)
• Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap [140.98628848491146]
  We introduce deconfounding scores, which induce better overlap without biasing the target of estimation. We show that deconfounding scores satisfy a zero-covariance condition that is identifiable in observed data. In particular, we show that this technique could be an attractive alternative to standard regularizations.
  arXiv  Detail & Related papers  (2021-04-12T18:50:11Z)
• Exploiting Sample Uncertainty for Domain Adaptive Person Re-Identification [137.9939571408506]
  We estimate and exploit the credibility of the assigned pseudo-label of each sample to alleviate the influence of noisy labels. Our uncertainty-guided optimization brings significant improvement and achieves the state-of-the-art performance on benchmark datasets.
  arXiv  Detail & Related papers  (2020-12-16T04:09:04Z)
• MatchGAN: A Self-Supervised Semi-Supervised Conditional Generative Adversarial Network [51.84251358009803]
  We present a novel self-supervised learning approach for conditional generative adversarial networks (GANs) under a semi-supervised setting. We perform augmentation by randomly sampling sensible labels from the label space of the few labelled examples available. Our method surpasses the baseline with only 20% of the labelled examples used to train the baseline.
  arXiv  Detail & Related papers  (2020-06-11T17:14:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.