Writing literature reviews with AI: principles, hurdles and some lessons learned

• URL: http://arxiv.org/abs/2603.20235v1
• Date: Sun, 08 Mar 2026 18:02:09 GMT
• Title: Writing literature reviews with AI: principles, hurdles and some lessons learned
• Abstract summary: We compare literature reviews produced with varying degrees of AI assistance.<n>Our comparison of six versions shows a series of pitfalls and suggests precautions necessary when using AI assistance to make a literature review.<n>Overall, AI can improve the span and quality of the review, but the gain of time is not as massive as one would expect, and a press-button strategy leaving AI to do the work is a recipe for disaster.
• Score: 0.15554979526414522
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We qualitatively compared literature reviews produced with varying degrees of AI assistance. The same LLM, given the same corpus of 280 papers but different selections, produced dramatically different reviews, from mainstream and politically neutral to critical and post-colonial, though neither orientation was intended. LLM outputs always appear at first glance to be well written, well informed and thought out, but closer reading reveals gaps, biases and lack of depth. Our comparison of six versions shows a series of pitfalls and suggests precautions necessary when using AI assistance to make a literature review. Main issues are: (1) The bias of ignorance (you do not know what you do not get) in the selection of relevant papers. (2) Alignment and digital sycophancy: commercial AI models slavishly take you further in the direction they understand you give them, reinforcing biases. (3) Mainstreaming: because of their statistical nature, LLM productions tend to favor mainstream perspectives and content; in our case there was only 20% overlap between paper selections by humans and the LLM. (4) Limited capacity for creative restructuring, with vague and ambiguous statements. (5) Lack of critical perspective, coming from distant reading and political correctness. Most pitfalls can be addressed by prompting, but only if the user knows the domain well enough to detect them. There is a paradox: producing a good AI-assisted review requires expertise that comes from reading the literature, which is precisely what AI was meant to reduce. Overall, AI can improve the span and quality of the review, but the gain of time is not as massive as one would expect, and a press-button strategy leaving AI to do the work is a recipe for disaster. We conclude with recommendations for those who write, or assess, such LLM-augmented reviews.

Related papers

• AI-Assisted Peer Review Across Research Communities: From Reviewer AI Policies to LLM Review Quality [39.33746637947531]
  We survey reviewer-facing AI policies across 111 leading AI/NLP conferences and medical journals.<n>We evaluate AI-generated peer reviews at ICLR 2026 and Nature Communications using a novel dataset.
  arXiv  Detail & Related papers  (2026-08-04T12:34:47Z)
• No Hidden Prompts Needed! You Can Game AI Peer Review with Presentation-Only Revisions [43.03725702558361]
  We study a harder and more policy-relevant failure mode: no hidden text, no prompt injection, and no changes to methods, experiments, figures, equations, proofs, or numerical results.<n>The attacker modifies only presentation-level content, such as the abstract, contribution framing, related work, discussion, and narrative structure.<n>Across three mainstream AI reviewers, adversarial repackaging achieves a 75.1% attack success rate and a mean score gain of +1.21/10.
  arXiv  Detail & Related papers  (2026-06-11T08:30:18Z)
• Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions [4.04252733799147]
  We study 20 computer architecture papers, with varying levels of submission lineage, to expose how well AI review aligns with human review.<n>To conduct the case study, we build a web UI-integrated tool, emphAI-Paper-Review, that generates structured AI review of a draft paper.
  arXiv  Detail & Related papers  (2026-05-31T05:05:26Z)
• On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists [113.03797263688519]
  Many scientists view AI reviewers as probabilistic systems without the expertise to evaluate research.<n>Existing evaluations of AI reviewers have focused on whether their verdicts match human verdicts.
  arXiv  Detail & Related papers  (2026-05-20T03:33:55Z)
• Stop Automating Peer Review Without Rigorous Evaluation [51.53099943385505]
  This position paper argues that today's AI systems should not be used to produce paper reviews.<n>We identify two critical issues: 1) AI reviewers exhibit a hivemind effect of excessive agreement within and across papers that reduces perspective diversity.<n>We argue that addressing the peer review crisis requires a science of peer review automation.
  arXiv  Detail & Related papers  (2026-05-04T22:41:04Z)
• Assessing LLMs in Art Contexts: Critique Generation and Theory of Mind Evaluation [0.3359875577705537]
  This study explores how large language models (LLMs) perform in two areas related to art.<n>For the critique generation part, we built a system that combines Noel Carroll's evaluative framework with a broad selection of art criticism theories.<n>These critiques were compared with those written by human experts in a Turing test-style evaluation.<n>In the second part, we introduced new simple ToM tasks based on situations involving interpretation, emotion, and moral tension.
  arXiv  Detail & Related papers  (2025-04-17T10:10:25Z)
• Could AI Trace and Explain the Origins of AI-Generated Images and Text? [53.11173194293537]
  AI-generated content is increasingly prevalent in the real world.<n> adversaries might exploit large multimodal models to create images that violate ethical or legal standards.<n>Paper reviewers may misuse large language models to generate reviews without genuine intellectual effort.
  arXiv  Detail & Related papers  (2025-04-05T20:51:54Z)
• Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing [55.2480439325792]
  This study systematically evaluations twelve state-of-the-art AI-text detectors using our AI-Polished-Text Evaluation dataset.<n>Our findings reveal that detectors frequently flag even minimally polished text as AI-generated, struggle to differentiate between degrees of AI involvement, and exhibit biases against older and smaller models.
  arXiv  Detail & Related papers  (2025-02-21T18:45:37Z)
• Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review [66.73247554182376]
  Large language models (LLMs) have led to their integration into peer review.<n>The unchecked adoption of LLMs poses significant risks to the integrity of the peer review system.<n>We show that manipulating 5% of the reviews could potentially cause 12% of the papers to lose their position in the top 30% rankings.
  arXiv  Detail & Related papers  (2024-12-02T16:55:03Z)
• LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing [106.45895712717612]
  Large language models (LLMs) have shown remarkable versatility in various generative tasks. This study focuses on the topic of LLMs assist NLP Researchers. To our knowledge, this is the first work to provide such a comprehensive analysis.
  arXiv  Detail & Related papers  (2024-06-24T01:30:22Z)
• Large Language Models' Detection of Political Orientation in Newspapers [0.0]
  Various methods have been developed to better understand newspapers' positioning. The advent of Large Language Models (LLM) hold disruptive potential to assist researchers and citizens alike. We compare how four widely employed LLMs rate the positioning of newspapers, and compare if their answers align with one another. Over a woldwide dataset, articles in newspapers are positioned strikingly differently by single LLMs, hinting to inconsistent training or excessive randomness in the algorithms.
  arXiv  Detail & Related papers  (2024-05-23T06:18:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.