GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization

• URL: http://arxiv.org/abs/2605.29107v2
• Date: Sat, 30 May 2026 18:12:24 GMT
• Title: GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization
• Abstract summary: We present GEO-Bench, a benchmark that evaluates GEO ranking-manipulation attacks under one protocol.<n>It unifies black-box prompt-based attacks (TAP, Zero-Shot), white-box gradient-based attacks (STS, RAF, StealthRank), and ten white-hat C-SEO strategies.<n>We score every method on five datasets against a fixed open-weight ranker.
• Score: 9.762893142689357
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) increasingly rank products, documents, and recommendations for user queries, which makes manipulating these rankings a growing concern for fairness and information integrity. Research on generative engine optimization (GEO) has produced many manipulation methods, but each is evaluated on its own dataset with its own metrics, so their relative strength and detectability stay unclear. We present GEO-Bench, a benchmark that evaluates GEO ranking-manipulation attacks under one protocol. It unifies black-box prompt-based attacks (TAP, Zero-Shot), white-box gradient-based attacks (STS, RAF, StealthRank), and ten white-hat C-SEO strategies. We score every method on five datasets against a fixed open-weight ranker (Llama-3.1-8B-Instruct), using metrics for both effectiveness (NRG, Success@α, Promote@α) and stealth (keyword violation rate, perplexity ratio). Our evaluation shows that effectiveness and stealth trade off across adversarial attacks, that black-box content rewriting matches or exceeds gradient-based attacks on rank promotion while producing more fluent text and can evade both keyword- and perplexity-based detection on some domains, and that the access model does not predict attack strength. By standardizing datasets, attack implementations, and metrics, GEO-Bench enables the first direct comparison across these attack paradigms and supports the development of detection methods.

Related papers

• The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage [71.8564105095189]
  We introduce N-Gram Coverage Attack, a membership inference attack that relies solely on text outputs from the target model.<n>We first demonstrate on a diverse set of existing benchmarks that N-Gram Coverage Attack outperforms other black-box methods.<n>We find that more recent models, such as GPT-4o, exhibit increased robustness to membership inference.
  arXiv  Detail & Related papers  (2025-08-13T08:35:16Z)
• CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Models [15.349703228157479]
  Retrieval-Augmented Generation (RAG) enhances large language models (LLMs) by incorporating external knowledge.<n>Existing poisoning methods for RAG systems have limitations, such as poor generalization and lack of fluency in adversarial texts.<n>We propose CPA-RAG, a black-box adversarial framework that generates query-relevant texts capable of manipulating the retrieval process to induce target answers.
  arXiv  Detail & Related papers  (2025-05-26T11:48:32Z)
• Checkpoint-GCG: Auditing and Attacking Fine-Tuning-Based Prompt Injection Defenses [10.08464073347558]
  We introduce Checkpoint-GCG, a white-box attack against fine-tuning-based defenses.<n>We show Checkpoint-GCG to achieve up to $96%$ attack success rate (ASR) against the strongest defense.
  arXiv  Detail & Related papers  (2025-05-21T16:43:17Z)
• Poison-RAG: Adversarial Data Poisoning Attacks on Retrieval-Augmented Generation in Recommender Systems [15.973406739758856]
  This study presents Poison-RAG, a framework for adversarial data poisoning attacks targeting recommender systems.<n>Using item metadata generated through a large language model (LLM) and embeddings derived via the OpenAI API, we explore the impact of adversarial poisoning attacks on provider-side.<n>Experiments conducted on the MovieLens dataset in a black-box setting reveal that local strategies improve manipulation effectiveness by up to 50%, while global strategies risk boosting already popular items.
  arXiv  Detail & Related papers  (2025-01-20T21:38:36Z)
• Attack-in-the-Chain: Bootstrapping Large Language Models for Attacks Against Black-box Neural Ranking Models [111.58315434849047]
  We introduce a novel ranking attack framework named Attack-in-the-Chain.<n>It tracks interactions between large language models (LLMs) and Neural ranking models (NRMs) based on chain-of-thought.<n> Empirical results on two web search benchmarks show the effectiveness of our method.
  arXiv  Detail & Related papers  (2024-12-25T04:03:09Z)
• EMPRA: Embedding Perturbation Rank Attack against Neural Ranking Models [20.106547084712922]
  Adversarial attacks seek to manipulate the ranking of documents, with the intention of exposing users to targeted content.<n>EMPRA manipulates sentence-level embeddings, guiding them towards pertinent context related to the query.<n>EMPRA generates adversarial texts that seamlessly integrate with the original content and remain imperceptible to humans.
  arXiv  Detail & Related papers  (2024-12-20T22:36:19Z)
• AdvQDet: Detecting Query-Based Adversarial Attacks with Adversarial Contrastive Prompt Tuning [93.77763753231338]
  Adversarial Contrastive Prompt Tuning (ACPT) is proposed to fine-tune the CLIP image encoder to extract similar embeddings for any two intermediate adversarial queries. We show that ACPT can detect 7 state-of-the-art query-based attacks with $>99%$ detection rate within 5 shots. We also show that ACPT is robust to 3 types of adaptive attacks.
  arXiv  Detail & Related papers  (2024-08-04T09:53:50Z)
• GE-AdvGAN: Improving the transferability of adversarial samples by gradient editing-based adversarial generative model [69.71629949747884]
  Adversarial generative models, such as Generative Adversarial Networks (GANs), are widely applied for generating various types of data. In this work, we propose a novel algorithm named GE-AdvGAN to enhance the transferability of adversarial samples.
  arXiv  Detail & Related papers  (2024-01-11T16:43:16Z)
• A Tale of HodgeRank and Spectral Method: Target Attack Against Rank Aggregation Is the Fixed Point of Adversarial Game [153.74942025516853]
  The intrinsic vulnerability of the rank aggregation methods is not well studied in the literature. In this paper, we focus on the purposeful adversary who desires to designate the aggregated results by modifying the pairwise data. The effectiveness of the suggested target attack strategies is demonstrated by a series of toy simulations and several real-world data experiments.
  arXiv  Detail & Related papers  (2022-09-13T05:59:02Z)
• RamBoAttack: A Robust Query Efficient Deep Neural Network Decision Exploit [9.93052896330371]
  We develop a robust query efficient attack capable of avoiding entrapment in a local minimum and misdirection from noisy gradients. The RamBoAttack is more robust to the different sample inputs available to an adversary and the targeted class.
  arXiv  Detail & Related papers  (2021-12-10T01:25:24Z)
• Poisoning Attack against Estimating from Pairwise Comparisons [140.9033911097995]
  Attackers have strong motivation and incentives to manipulate the ranking list. Data poisoning attacks on pairwise ranking algorithms can be formalized as the dynamic and static games between the ranker and the attacker. We propose two efficient poisoning attack algorithms and establish the associated theoretical guarantees.
  arXiv  Detail & Related papers  (2021-07-05T08:16:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.