How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews

• URL: http://arxiv.org/abs/2604.27790v1
• Date: Thu, 30 Apr 2026 12:32:21 GMT
• Title: How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviews
• Abstract summary: We compare the search results returned by Google's search engine, the accompanying AI Overview (AIO), and Gemini Flash 2.5 for each query.<n>For 51.5% of representative, real-user queries, AIOs are generated, and are displayed above the organic search results.<n>Traditional Google search is significantly more likely to retrieve information from popular or institutional websites in government or education.<n>Generative search engines are significantly more likely to retrieve Google-owned content.
• Score: 5.313243639890497
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative AI is being increasingly integrated into web search for the convenience it provides users. In this work, we aim to understand how generative AI disrupts web search by retrieving and presenting the information and sources differently from traditional search engines. We introduce a public benchmark dataset of 11,500 user queries to support our study and future research of generative search. We compare the search results returned by Google's search engine, the accompanying AI Overview (AIO), and Gemini Flash 2.5 for each query. We have made several key findings. First, we find that for 51.5\% of representative, real-user queries, AIOs are generated, and are displayed above the organic search results. Controversial questions frequently result in an AIO. Second, we show that the retrieved sources are substantially different for each search engine (<0.2 average Jaccard similarity). Traditional Google search is significantly more likely to retrieve information from popular or institutional websites in government or education, while generative search engines are significantly more likely to retrieve Google-owned content. Third, we observe that websites that block Google's AI crawler are significantly less likely to be retrieved by AIOs, despite having access to the content. Finally, AIOs are less consistent when processing two runs of the same query, and are less robust to minor query edits. Our findings have important implications for understanding how generative search impacts website visibility, the effectiveness of generative engine optimization techniques, and the information users receive. We call for revenue frameworks to foster a sustainable and mutually beneficial ecosystem for publishers and generative search providers.

Related papers

• AI in Search Reduces Publisher Referrals Without Improving User Experience: Experimental Evidence [2.303636351420161]
  We estimate the causal effects of AI Overviews and AI Mode on user behavior, perceptions, and publisher traffic on Google Search.<n>We show that removing AI Overviews and AI Mode increases click-through rates to publishers, while an AI Mode-only experience reduces click-through rates and erodes user experience and trust in information found on Google.
  arXiv  Detail & Related papers  (2026-08-18T22:08:10Z)
• Measuring Google AI Overviews: Activation, Source Quality, Claim Fidelity, and Publisher Impact [10.145178533574695]
  Google AI Overviews (AIOs) are arguably the most widely encountered deployment of generative AI, reaching over 2 billion users who may not realize the answers they see are AI-generated.<n>We present a large-scale longitudinal measurement study, issuing 55,393 trending queries across 19 topical categories over a 40-day window (March 13 - April 21, 2026).<n>We report four main findings. First, overall AIO activation is 13.7%, rising to 64.7% for question-form queries, while politically sensitive topics see markedly lower rates.
  arXiv  Detail & Related papers  (2026-05-13T18:34:39Z)
• AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery [55.70879973230979]
  AutoResearchBench is a benchmark for autonomous scientific literature discovery.<n>Compared to previous benchmarks on agentic web browsing, AutoResearchBench is research-oriented.<n>Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research.
  arXiv  Detail & Related papers  (2026-04-28T06:05:17Z)
• Navigating the Shift: A Comparative Analysis of Web Search and Generative AI Response Generation [13.959899706228176]
  The rise of generative AI as a primary information source presents a paradigm shift from traditional web search.<n>This paper presents a large-scale empirical study quantifying the fundamental differences between the results returned by Google Search and leading generative AI services.
  arXiv  Detail & Related papers  (2026-01-23T16:06:18Z)
• Characterizing Web Search in The Age of Generative AI [7.059953211629231]
  We compare Google, a traditional web search engine, with four generative search engines from two providers (Google and OpenAI)<n>Generative search engines vary in the degree to which they rely on internal knowledge contained within the model parameters v.s. external knowledge retrieved from the web.<n>Our results highlight the need for revisiting evaluation criteria for web search in the age of Generative AI.
  arXiv  Detail & Related papers  (2025-10-13T16:04:03Z)
• Generative Engine Optimization: How to Dominate AI Search [13.959899706228176]
  Generative AI-powered search engines like ChatGPT, Perplexity, and Gemini are reshaping information retrieval.<n>This paper presents a comprehensive analysis of AI Search and traditional web search (Google)<n>Our key findings reveal that AI Search exhibit a systematic and overwhelming bias towards Earned media (third-party, authoritative sources) over Brand-owned and Social content.
  arXiv  Detail & Related papers  (2025-09-10T18:29:18Z)
• From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agents [96.65646344634524]
  Large Language Models (LLMs), endowed with reasoning and agentic capabilities, are ushering in a new paradigm termed Agentic Deep Research.<n>We trace the evolution from static web search to interactive, agent-based systems that plan, explore, and learn.<n>We demonstrate that Agentic Deep Research not only significantly outperforms existing approaches, but is also poised to become the dominant paradigm for future information seeking.
  arXiv  Detail & Related papers  (2025-06-23T17:27:19Z)
• MindSearch: Mimicking Human Minds Elicits Deep AI Searcher [50.68599514830046]
  We introduce MindSearch to mimic the human minds in web information seeking and integration.<n>The framework can be instantiated by a simple yet effective LLM-based multi-agent framework.<n> MindSearch demonstrates significant improvement in the response quality in terms of depth and breadth.
  arXiv  Detail & Related papers  (2024-07-29T17:12:40Z)
• Tree Search for Language Model Agents [73.97960454223164]
  We propose an inference-time search algorithm for LM agents to perform exploration and multi-step planning in interactive web environments.<n>Our approach is a form of best-first tree search that operates within the actual environment space.<n>It is the first tree search algorithm for LM agents that shows effectiveness on realistic web tasks.
  arXiv  Detail & Related papers  (2024-07-01T17:07:55Z)
• Evaluating Verifiability in Generative Search Engines [70.59477647085387]
  Generative search engines directly generate responses to user queries, along with in-line citations. We conduct human evaluation to audit four popular generative search engines. We find that responses from existing generative search engines are fluent and appear informative, but frequently contain unsupported statements and inaccurate citations.
  arXiv  Detail & Related papers  (2023-04-19T17:56:12Z)
• NeuralSearchX: Serving a Multi-billion-parameter Reranker for Multilingual Metasearch at a Low Cost [4.186775801993103]
  We describe NeuralSearchX, a metasearch engine based on a multi-purpose large reranking model to merge results and highlight sentences. We show that our design choices led to a much cost-effective system with competitive QPS while having close to state-of-the-art results on a wide range of public benchmarks.
  arXiv  Detail & Related papers  (2022-10-26T16:36:53Z)
• Exposing Query Identification for Search Transparency [69.06545074617685]
  We explore the feasibility of approximate exposing query identification (EQI) as a retrieval task by reversing the role of queries and documents in two classes of search systems. We derive an evaluation metric to measure the quality of a ranking of exposing queries, as well as conducting an empirical analysis focusing on various practical aspects of approximate EQI.
  arXiv  Detail & Related papers  (2021-10-14T20:19:27Z)
• Search Engine Similarity Analysis: A Combined Content and Rankings Approach [6.69087470775851]
  We present an analysis of the affinity of the two major search engines, Google and Bing, along with DuckDuckGo. We developed a new similarity metric that leverages both the content and the ranking of search responses. We found that Google stands apart, but Bing and DuckDuckGo are largely indistinguishable from each other.
  arXiv  Detail & Related papers  (2020-11-01T23:57:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.