Manalyzer: End-to-end Automated Meta-analysis with Multi-agent System

• URL: http://arxiv.org/abs/2505.20310v1
• Date: Thu, 22 May 2025 07:25:31 GMT
• Title: Manalyzer: End-to-end Automated Meta-analysis with Multi-agent System
• Authors: Wanghan Xu, Wenlong Zhang, Fenghua Ling, Ben Fei, Yusong Hu, Fangxuan Ren, Jintai Lin, Wanli Ouyang, Lei Bai,
• Abstract summary: Meta-analysis is a systematic research methodology that synthesizes data from multiple existing studies to derive comprehensive conclusions.<n>Traditional meta-analysis involves a complex multi-stage pipeline including literature retrieval, paper screening, and data extraction.<n>We propose a multi-agent system, Manalyzer, which achieves end-to-end automated meta-analysis through tool calls.
• Score: 48.093356587573666
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Meta-analysis is a systematic research methodology that synthesizes data from multiple existing studies to derive comprehensive conclusions. This approach not only mitigates limitations inherent in individual studies but also facilitates novel discoveries through integrated data analysis. Traditional meta-analysis involves a complex multi-stage pipeline including literature retrieval, paper screening, and data extraction, which demands substantial human effort and time. However, while LLM-based methods can accelerate certain stages, they still face significant challenges, such as hallucinations in paper screening and data extraction. In this paper, we propose a multi-agent system, Manalyzer, which achieves end-to-end automated meta-analysis through tool calls. The hybrid review, hierarchical extraction, self-proving, and feedback checking strategies implemented in Manalyzer significantly alleviate these two hallucinations. To comprehensively evaluate the performance of meta-analysis, we construct a new benchmark comprising 729 papers across 3 domains, encompassing text, image, and table modalities, with over 10,000 data points. Extensive experiments demonstrate that Manalyzer achieves significant performance improvements over the LLM baseline in multi meta-analysis tasks. Project page: https://black-yt.github.io/meta-analysis-page/ .

Related papers

• MOLE: Metadata Extraction and Validation in Scientific Papers Using LLMs [54.5729817345543]
  MOLE is a framework that automatically extracts metadata attributes from scientific papers covering datasets of languages other than Arabic.<n>Our methodology processes entire documents across multiple input formats and incorporates robust validation mechanisms for consistent output.
  arXiv  Detail & Related papers  (2025-05-26T10:31:26Z)
• IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis [60.32962597618861]
  IDA-Bench is a novel benchmark evaluating large language models in multi-round interactive scenarios.<n>Agent performance is judged by comparing its final numerical output to the human-derived baseline.<n>Even state-of-the-art coding agents (like Claude-3.7-thinking) succeed on 50% of the tasks, highlighting limitations not evident in single-turn tests.
  arXiv  Detail & Related papers  (2025-05-23T09:37:52Z)
• Flowco: Rethinking Data Analysis in the Age of LLMs [2.1874189959020427]
  Large language models (LLMs) are now capable of generating such code for simple, routine analyses.<n>LLMs promise to democratize data science by enabling those with limited programming expertise to conduct data analyses.<n>Analysts in many real-world settings must often exercise fine-grained control over specific analysis steps.<n>This paper introduces Flowco, a new mixed-initiative system to address these challenges.
  arXiv  Detail & Related papers  (2025-04-18T19:01:27Z)
• Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMs [32.48924329288906]
  This study presents a semi-automated approach for literature analysis that accelerates data extraction using LLMs.<n>It automatically identifies relevant arXiv papers, extracts experimental results and related attributes, and organizes them into a structured dataset, LLMEvalDB.<n>We then conduct an automated literature analysis of frontier LLMs, reducing the effort of paper surveying and data extraction by more than 93% compared to manual approaches.
  arXiv  Detail & Related papers  (2025-02-26T03:56:34Z)
• MIMDE: Exploring the Use of Synthetic vs Human Data for Evaluating Multi-Insight Multi-Document Extraction Tasks [0.0]
  We define a set of tasks, Multi-Insight Multi-Document Extraction (MIMDE) tasks.<n>This task is fundamental to many practical applications, from analyzing survey responses to processing medical records.<n>We introduce a novel set of complementary human and synthetic datasets to examine the potential of synthetic data for evaluation.
  arXiv  Detail & Related papers  (2024-11-29T13:24:10Z)
• Empowering Meta-Analysis: Leveraging Large Language Models for Scientific Synthesis [7.059964549363294]
  This study investigates the automation of meta-analysis in scientific documents using large language models (LLMs) Our research introduces a novel approach that fine-tunes the LLM on extensive scientific datasets to address challenges in big data handling and structured data extraction.
  arXiv  Detail & Related papers  (2024-11-16T20:18:57Z)
• Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles [136.84278943588652]
  We propose a new task of summarizing diverse information encountered in multiple news articles encompassing the same event. To facilitate this task, we outlined a data collection schema for identifying diverse information and curated a dataset named DiverseSumm. The dataset includes 245 news stories, with each story comprising 10 news articles and paired with a human-validated reference.
  arXiv  Detail & Related papers  (2023-09-17T20:28:17Z)
• Automated Meta-Analysis: A Causal Learning Perspective [30.746257517698133]
  We work toward automating meta-analysis with a focus on controlling for risks of bias. We first extract information from scientific publications written in natural language. From a novel causal learning perspective, we propose to frame automated meta-analysis as a multiple-causal-inference problem.
  arXiv  Detail & Related papers  (2021-04-09T23:07:07Z)
• Alchemy: A structured task distribution for meta-reinforcement learning [52.75769317355963]
  We introduce a new benchmark for meta-RL research, which combines structural richness with structural transparency. Alchemy is a 3D video game, which involves a latent causal structure that is resampled procedurally from episode to episode. We evaluate a pair of powerful RL agents on Alchemy and present an in-depth analysis of one of these agents.
  arXiv  Detail & Related papers  (2021-02-04T23:40:44Z)
• Transformer-based Multi-Aspect Modeling for Multi-Aspect Multi-Sentiment Analysis [56.893393134328996]
  We propose a novel Transformer-based Multi-aspect Modeling scheme (TMM), which can capture potential relations between multiple aspects and simultaneously detect the sentiment of all aspects in a sentence. Our method achieves noticeable improvements compared with strong baselines such as BERT and RoBERTa.
  arXiv  Detail & Related papers  (2020-11-01T11:06:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.