AI Overviews in Academic Search: Evaluating AI-generated Summaries of Search Results in a Domain-specific Search Engine

• URL: http://arxiv.org/abs/2607.03421v1
• Date: Fri, 03 Jul 2026 15:35:08 GMT
• Title: AI Overviews in Academic Search: Evaluating AI-generated Summaries of Search Results in a Domain-specific Search Engine
• Abstract summary: We evaluate AI-generated summaries as a support feature in an academic search engine for social science information.<n>We find non-significant trends favoring AI summaries for subjective workload, perceived usefulness, satisfaction, and decision-making confidence.
• Score: 4.04785724664289
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating search engine results pages (SERPs) to assess result relevance is a demanding step in academic search. In a formative mixed-methods design study, we examine AI-generated SERP-level summaries as a support feature in an academic search engine for social science information. First, we manually evaluated summaries of the top five results for 10 queries using two general-purpose models, one commercial and one open, deriving an exploratory six-category error taxonomy and five safeguards for scholarly deployment. We then conducted a within-subjects user study (n = 30) comparing interfaces with and without AI summaries. Confirmatory analyses showed consistent but non-significant trends favoring AI summaries for subjective workload, perceived usefulness, satisfaction, and decision-making confidence. Exploratory analyses suggested lower mental demand, with frustration also tending to be lower. Behaviorally, participants rarely expanded the summaries and descriptively made slightly fewer result clicks and query reformulations when summaries were available. Drawing on Information Foraging Theory and participant feedback, we suggest that AI summaries may concentrate SERP-level information scent to support early triage. Overall, the findings indicate that SERP-level AI summaries are a context- and user-dependent aid rather than a universal improvement, while contributing an error taxonomy, safeguard-aware deployment guidance, and concrete design implications for scholarly search.

Related papers

• Cost-efficient generative AI summarization for scalable automated essay scoring in educational assessment [0.40611352512781873]
  This study proposes a generative AI-assisted summarization framework to improve long-form essay representation.<n>We generate controlled-length summaries with three GPT-5 variants (GPT-5, GPT-5 mini, and GPT-5 nano) and use them as inputs for downstream AES models.<n>To preserve original writing signals, handcrafted linguistic features extracted from full essays are integrated with summary representations.
  arXiv  Detail & Related papers  (2026-07-17T10:39:58Z)
• AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery [55.70879973230979]
  AutoResearchBench is a benchmark for autonomous scientific literature discovery.<n>Compared to previous benchmarks on agentic web browsing, AutoResearchBench is research-oriented.<n>Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research.
  arXiv  Detail & Related papers  (2026-04-28T06:05:17Z)
• The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research [56.80927148740585]
  We address the challenges of scalability and rigor by flipping the dynamic and developing AI agents as research evaluators.<n>We use mechanistic interpretability research as a testbed, build standardized research output, and develop MechEvalAgent.<n>Our work demonstrates the potential of AI agents to transform research evaluation and pave the way for rigorous scientific practices.
  arXiv  Detail & Related papers  (2026-02-05T19:00:02Z)
• Let the Barbarians In: How AI Can Accelerate Systems Performance Research [80.43506848683633]
  We term this iterative cycle of generation, evaluation, and refinement AI-Driven Research for Systems.<n>We demonstrate that ADRS-generated solutions can match or even outperform human state-of-the-art designs.
  arXiv  Detail & Related papers  (2025-12-16T18:51:23Z)
• AI summaries in online search influence users' attitudes [3.459756369056329]
  This study examined how AI-generated summaries affect how users think about different issues.<n>Users perceived the AI summaries more useful when it emphasized health harms versus benefits.<n>These findings suggest that AI-generated search summaries can significantly shape public perceptions.
  arXiv  Detail & Related papers  (2025-11-27T23:45:19Z)
• Automated Classification of Tutors' Dialogue Acts Using Generative AI: A Case Study Using the CIMA Corpus [10.325932865188514]
  This case study uses the open-source CIMA corpus, in which tutors' responses are pre-annotated into four DA categories.<n>Results show that GPT-4 achieved 80% accuracy, a weighted F1-score of 0.81, and a Cohen's Kappa of 0.74.
  arXiv  Detail & Related papers  (2025-09-11T03:36:03Z)
• Reasoning-enhanced Query Understanding through Decomposition and Interpretation [87.56450566014625]
  ReDI is a Reasoning-enhanced approach for query understanding through Decomposition and Interpretation.<n>We compiled a large-scale dataset of real-world complex queries from a major search engine.<n> Experiments on BRIGHT and BEIR demonstrate that ReDI consistently surpasses strong baselines in both sparse and dense retrieval paradigms.
  arXiv  Detail & Related papers  (2025-09-08T10:58:42Z)
• BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent [74.10138164281618]
  BrowseComp-Plus is a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus.<n>This benchmark allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods.
  arXiv  Detail & Related papers  (2025-08-08T17:55:11Z)
• Are Large Language Models Ready for Business Integration? A Study on Generative AI Adoption [0.6144680854063939]
  This research examines the readiness of other Large Language Models (LLMs) such as Google Gemini for potential business applications.<n>A dataset of 42,654 reviews from distinct Disneyland branches was employed.<n>Results presented a spectrum of responses, including 75% successful simplifications, 25% errors, and instances of model self-reference.
  arXiv  Detail & Related papers  (2025-01-28T21:01:22Z)
• A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [51.26815896167173]
  We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
  arXiv  Detail & Related papers  (2024-02-20T11:28:50Z)
• Automatic Text Summarization Methods: A Comprehensive Review [1.6114012813668934]
  This study provides a detailed analysis of text summarization concepts such as summarization approaches, techniques used, standard datasets, evaluation metrics and future scopes for research.
  arXiv  Detail & Related papers  (2022-03-03T10:45:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.