A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence

• URL: http://arxiv.org/abs/2402.12928v6
• Date: Sat, 06 Sep 2025 03:51:53 GMT
• Title: A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence
• Authors: Penghai Zhao, Xin Zhang, Jiayue Cao, Ming-Ming Cheng, Jian Yang, Xiang Li,
• Abstract summary: We present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions.<n>Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices.<n>Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization.
• Score: 51.26815896167173
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The rapid growth of research in Pattern Analysis and Machine Intelligence (PAMI) has rendered literature reviews essential for consolidating and interpreting knowledge across its many subfields. In this work, we present a comprehensive tertiary analysis of PAMI reviews along three complementary dimensions: (i) identifying structural and statistical regularities in existing surveys; (ii) developing quantitative strategies that help researchers navigate and prioritize within the expanding review corpus; and (iii) critically assessing emerging AI-generated review systems. To support this study, we construct RiPAMI, a large-scale database containing more than 3,000 review articles, and combine narrative synthesis with statistical analysis to capture structural and content-level features. Our analyses reveal distinctive organizational patterns as well as persistent gaps in current review practices. Building on these insights, we propose practical, article-level strategies for indicator-guided navigation that move beyond simple citation counts. Finally, our evaluation of state-of-the-art AI-generated reviews indicates encouraging advances in coherence and organization, yet also highlights enduring weaknesses in reference retrieval, coverage of recent work, and the incorporation of visual elements. Together, these findings provide both a critical appraisal of existing review practices and a forward-looking perspective on how AI-generated reviews can evolve into trustworthy, customizable, and transformative complements to traditional human-authored surveys.

Related papers

• Position: Thematic Analysis of Unstructured Clinical Transcripts with Large Language Models [5.398283020969301]
  Large language models (LLMs) can support thematic analysis of unstructured clinical transcripts.<n>Existing evaluation methods vary widely, hindering progress and preventing meaningful benchmarking across studies.<n>We propose an evaluation framework centered on three dimensions: validity, reliability, and interpretability.
  arXiv  Detail & Related papers  (2025-09-18T04:02:00Z)
• Beyond "Not Novel Enough": Enriching Scholarly Critique with LLM-Assisted Feedback [81.0031690510116]
  We present a structured approach for automated novelty evaluation that models expert reviewer behavior through three stages.<n>Our method is informed by a large scale analysis of human written novelty reviews.<n> Evaluated on 182 ICLR 2025 submissions, the approach achieves 86.5% alignment with human reasoning and 75.3% agreement on novelty conclusions.
  arXiv  Detail & Related papers  (2025-08-14T16:18:37Z)
• From Multimodal Perception to Strategic Reasoning: A Survey on AI-Generated Game Commentary [9.045787191833822]
  We present a novel taxonomy focused on three core commentator capabilities: Live Observation, Strategic Analysis, and Historical Recall.<n>We provide an in-depth review of state-of-the-art methods, datasets, and evaluation metrics across various game genres.
  arXiv  Detail & Related papers  (2025-06-17T07:04:51Z)
• Trends and Challenges in Authorship Analysis: A Review of ML, DL, and LLM Approaches [1.8686807993563161]
  Authorship analysis plays an important role in diverse domains, including forensic linguistics, academia, cybersecurity, and digital content authentication.<n>This paper presents a systematic literature review on two key sub-tasks of authorship analysis; Author Attribution and Author Verification.
  arXiv  Detail & Related papers  (2025-05-21T12:06:08Z)
• In-depth Research Impact Summarization through Fine-Grained Temporal Citation Analysis [52.42612945266194]
  We propose a new task: generating nuanced, expressive, and time-aware impact summaries.<n>We show that these summaries capture both praise (confirmation citations) and critique (correction citations) through the evolution of fine-grained citation intents.
  arXiv  Detail & Related papers  (2025-05-20T19:11:06Z)
• Identifying Aspects in Peer Reviews [61.374437855024844]
  We develop a data-driven schema for deriving fine-grained aspects from a corpus of peer reviews. We introduce a dataset of peer reviews augmented with aspects and show how it can be used for community-level review analysis.
  arXiv  Detail & Related papers  (2025-04-09T14:14:42Z)
• Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework [61.38174427966444]
  Large Language Models (LLMs) are being used more and more extensively for automated evaluation in various scenarios.<n>Previous studies have attempted to fine-tune open-source LLMs to replicate the evaluation explanations and judgments of powerful proprietary models.<n>We propose a novel evaluation framework, ARJudge, that adaptively formulates evaluation criteria and synthesizes both text-based and code-driven analyses.
  arXiv  Detail & Related papers  (2025-02-26T06:31:45Z)
• Generative Adversarial Reviews: When LLMs Become the Critic [1.2430809884830318]
  We introduce Generative Agent Reviewers (GAR), leveraging LLM-empowered agents to simulate faithful peer reviewers. Central to this approach is a graph-based representation of manuscripts, condensing content and logically organizing information. Our experiments demonstrate that GAR performs comparably to human reviewers in providing detailed feedback and predicting paper outcomes.
  arXiv  Detail & Related papers  (2024-12-09T06:58:17Z)
• Automating Bibliometric Analysis with Sentence Transformers and Retrieval-Augmented Generation (RAG): A Pilot Study in Semantic and Contextual Search for Customized Literature Characterization for High-Impact Urban Research [2.1728621449144763]
  Bibliometric analysis is essential for understanding research trends, scope, and impact in urban science. Traditional methods, relying on keyword searches, often fail to uncover valuable insights not explicitly stated in article titles or keywords. We leverage Generative AI models, specifically transformers and Retrieval-Augmented Generation (RAG), to automate and enhance bibliometric analysis.
  arXiv  Detail & Related papers  (2024-10-08T05:13:27Z)
• STRICTA: Structured Reasoning in Critical Text Assessment for Peer Review and Beyond [68.47402386668846]
  We introduce Structured Reasoning In Critical Text Assessment (STRICTA) to model text assessment as an explicit, step-wise reasoning process.<n>STRICTA breaks down the assessment into a graph of interconnected reasoning steps drawing on causality theory.<n>We apply STRICTA to a dataset of over 4000 reasoning steps from roughly 40 biomedical experts on more than 20 papers.
  arXiv  Detail & Related papers  (2024-09-09T06:55:37Z)
• LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing [106.45895712717612]
  Large language models (LLMs) have shown remarkable versatility in various generative tasks. This study focuses on the topic of LLMs assist NLP Researchers. To our knowledge, this is the first work to provide such a comprehensive analysis.
  arXiv  Detail & Related papers  (2024-06-24T01:30:22Z)
• AgentReview: Exploring Peer Review Dynamics with LLM Agents [13.826819101545926]
  We introduce AgentReview, the first large language model (LLM) based peer review simulation framework. Our study reveals significant insights, including a notable 37.1% variation in paper decisions due to reviewers' biases.
  arXiv  Detail & Related papers  (2024-06-18T15:22:12Z)
• RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance [0.8089605035945486]
  We propose RelevAI-Reviewer, an automatic system that conceptualizes the task of survey paper review as a classification problem. We introduce a novel dataset comprised of 25,164 instances. Each instance contains one prompt and four candidate papers, each varying in relevance to the prompt. We develop a machine learning (ML) model capable of determining the relevance of each paper and identifying the most pertinent one.
  arXiv  Detail & Related papers  (2024-06-13T06:42:32Z)
• ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models [56.08917291606421]
  ResearchAgent is an AI-based system for ideation and operationalization of novel work. ResearchAgent automatically defines novel problems, proposes methods and designs experiments, while iteratively refining them. We experimentally validate our ResearchAgent on scientific publications across multiple disciplines.
  arXiv  Detail & Related papers  (2024-04-11T13:36:29Z)
• CASIMIR: A Corpus of Scientific Articles enhanced with Multiple Author-Integrated Revisions [7.503795054002406]
  We propose an original textual resource on the revision step of the writing process of scientific articles. This new dataset, called CASIMIR, contains the multiple revised versions of 15,646 scientific articles from OpenReview, along with their peer reviews.
  arXiv  Detail & Related papers  (2024-03-01T03:07:32Z)
• Scientific Opinion Summarization: Paper Meta-review Generation Dataset, Methods, and Evaluation [55.00687185394986]
  We propose the task of scientific opinion summarization, where research paper reviews are synthesized into meta-reviews. We introduce the ORSUM dataset covering 15,062 paper meta-reviews and 57,536 paper reviews from 47 conferences. Our experiments show that (1) human-written summaries do not always satisfy all necessary criteria such as depth of discussion, and identifying consensus and controversy for the specific domain, and (2) the combination of task decomposition and iterative self-refinement shows strong potential for enhancing the opinions.
  arXiv  Detail & Related papers  (2023-05-24T02:33:35Z)
• Hierarchical Catalogue Generation for Literature Review: A Benchmark [36.22298354302282]
  We construct a novel English Hierarchical Catalogues of Literature Reviews dataset with 7.6k literature review catalogues and 389k reference papers. To accurately assess the model performance, we design two evaluation metrics for informativeness and similarity to ground truth from semantics and structure.
  arXiv  Detail & Related papers  (2023-04-07T07:13:35Z)
• Artificial intelligence technologies to support research assessment: A review [10.203602318836444]
  This literature review identifies indicators that associate with higher impact or higher quality research from article text. It includes studies that used machine learning techniques to predict citation counts or quality scores for journal articles or conference papers.
  arXiv  Detail & Related papers  (2022-12-11T06:58:39Z)
• NLPeer: A Unified Resource for the Computational Study of Peer Review [58.71736531356398]
  We introduce NLPeer -- the first ethically sourced multidomain corpus of more than 5k papers and 11k review reports from five different venues. We augment previous peer review datasets to include parsed and structured paper representations, rich metadata and versioning information. Our work paves the path towards systematic, multi-faceted, evidence-based study of peer review in NLP and beyond.
  arXiv  Detail & Related papers  (2022-11-12T12:29:38Z)
• Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach [77.61131357420201]
  We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs) We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date. We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
  arXiv  Detail & Related papers  (2022-11-07T16:19:42Z)
• State-of-the-art generalisation research in NLP: A taxonomy and review [87.1541712509283]
  We present a taxonomy for characterising and understanding generalisation research in NLP. Our taxonomy is based on an extensive literature review of generalisation research. We use our taxonomy to classify over 400 papers that test generalisation.
  arXiv  Detail & Related papers  (2022-10-06T16:53:33Z)
• An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics [33.655334920298856]
  We provide a comprehensive overview of the research on long document summarization. We conduct an empirical analysis to broaden the perspective on current research progress.
  arXiv  Detail & Related papers  (2022-07-03T02:57:22Z)
• Ranking Scientific Papers Using Preference Learning [48.78161994501516]
  We cast it as a paper ranking problem based on peer review texts and reviewer scores. We introduce a novel, multi-faceted generic evaluation framework for making final decisions based on peer reviews.
  arXiv  Detail & Related papers  (2021-09-02T19:41:47Z)
• A Systematic Literature Review of Empiricism and Norms of Reporting in Computing Education Research Literature [4.339510167603376]
  The goal of this study is to characterize the reporting of empiricism in Computing Education Research (CER) literature. We conducted an SLR of 427 papers published during 2014 and 2015 in five CER venues. Over 80% of papers had some form of empirical evaluation.
  arXiv  Detail & Related papers  (2021-07-02T16:37:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.