ADVISE: AI-accelerated Design of Evidence Synthesis for Global Development

• URL: http://arxiv.org/abs/2305.01145v1
• Date: Tue, 2 May 2023 01:29:53 GMT
• Title: ADVISE: AI-accelerated Design of Evidence Synthesis for Global Development
• Authors: Kristen M. Edwards, Binyang Song, Jaron Porciello, Mark Engelbert, Carolyn Huang, Faez Ahmed
• Abstract summary: This study develops an AI agent based on a bidirectional encoder representations from transformers (BERT) model. We explore the effectiveness of the human-AI hybrid team in accelerating the evidence synthesis process. Results show that incorporating the BERT-based AI agent into the human team can reduce the human screening effort by 68.5%.
• Score: 2.6293574825904624
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When designing evidence-based policies and programs, decision-makers must distill key information from a vast and rapidly growing literature base. Identifying relevant literature from raw search results is time and resource intensive, and is often done by manual screening. In this study, we develop an AI agent based on a bidirectional encoder representations from transformers (BERT) model and incorporate it into a human team designing an evidence synthesis product for global development. We explore the effectiveness of the human-AI hybrid team in accelerating the evidence synthesis process. To further improve team efficiency, we enhance the human-AI hybrid team through active learning (AL). Specifically, we explore different sampling strategies, including random sampling, least confidence (LC) sampling, and highest priority (HP) sampling, to study their influence on the collaborative screening process. Results show that incorporating the BERT-based AI agent into the human team can reduce the human screening effort by 68.5% compared to the case of no AI assistance and by 16.8% compared to the case of using a support vector machine (SVM)-based AI agent for identifying 80% of all relevant documents. When we apply the HP sampling strategy for AL, the human screening effort can be reduced even more: by 78.3% for identifying 80% of all relevant documents compared to no AI assistance. We apply the AL-enhanced human-AI hybrid teaming workflow in the design process of three evidence gap maps (EGMs) for USAID and find it to be highly effective. These findings demonstrate how AI can accelerate the development of evidence synthesis products and promote timely evidence-based decision making in global development in a human-AI hybrid teaming context.

Related papers

• STELLA: Self-Evolving LLM Agent for Biomedical Research [40.841136388072385]
  We introduce STELLA, a self-evolving AI agent designed to overcome limitations.<n> STELLA employs a multi-agent architecture that autonomously improves its own capabilities.<n>We demonstrate that STELLA achieves state-of-the-art accuracy on a suite of biomedical benchmarks.
  arXiv  Detail & Related papers  (2025-07-01T20:52:01Z)
• Fine-Tuning and Prompt Engineering of LLMs, for the Creation of Multi-Agent AI for Addressing Sustainable Protein Production Challenges [38.405196084093454]
  We present a proof-of-concept multi-agent Artificial Intelligence framework to support sustainable protein production research.<n>A literature search agent retrieves relevant scientific literature on microbial protein production for a specified microbial strain.<n>An information extraction agent processes the retrieved content to extract relevant biological and chemical information.
  arXiv  Detail & Related papers  (2025-06-25T16:37:46Z)
• Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities [117.49715661395294]
  Data structurization can play a promising role by transforming intricate and disorganized data into well-structured forms.<n>This survey presents a first systematic review of how graphs can empower AI agents.
  arXiv  Detail & Related papers  (2025-06-22T12:59:12Z)
• Model Cards for AI Teammates: Comparing Human-AI Team Familiarization Methods for High-Stakes Environments [0.0]
  Three methods of familiarizing a human with an artificial intelligence teammate were studied.<n>The most valuable information about the agent included details of its decision-making algorithms and its relative strengths and weaknesses compared to the human.<n>We recommend a human-AI team familiarization method that combines AI documentation, structured in-situ training, and exploratory interaction.
  arXiv  Detail & Related papers  (2025-05-19T23:19:16Z)
• Assessing Generative AI value in a public sector context: evidence from a field experiment [0.0]
  We find mixed evidence for two types of composite tasks related to document understanding and data analysis. For the Documents task, the treatment group using Gen AI had a 17% improvement in answer quality scores and a 34% improvement in task completion time compared to a control group. For the Data task, we find the Gen AI treatment group experienced a 12% reduction in quality scores and no significant difference in mean completion time compared to the control group.
  arXiv  Detail & Related papers  (2025-02-13T16:43:32Z)
• Multi-Agent Sampling: Scaling Inference Compute for Data Synthesis with Tree Search-Based Agentic Collaboration [81.45763823762682]
  This work aims to bridge the gap by investigating the problem of data synthesis through multi-agent sampling. We introduce Tree Search-based Orchestrated Agents(TOA), where the workflow evolves iteratively during the sequential sampling process. Our experiments on alignment, machine translation, and mathematical reasoning demonstrate that multi-agent sampling significantly outperforms single-agent sampling as inference compute scales.
  arXiv  Detail & Related papers  (2024-12-22T15:16:44Z)
• Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration [51.452664740963066]
  Collaborative Gym is a framework enabling asynchronous, tripartite interaction among agents, humans, and task environments. We instantiate Co-Gym with three representative tasks in both simulated and real-world conditions. Our findings reveal that collaborative agents consistently outperform their fully autonomous counterparts in task performance.
  arXiv  Detail & Related papers  (2024-12-20T09:21:15Z)
• Generative AI in Health Economics and Outcomes Research: A Taxonomy of Key Definitions and Emerging Applications, an ISPOR Working Group Report [12.204470166456561]
  Generative AI shows significant potential in health economics and outcomes research (HEOR) Generative AI shows significant potential in HEOR, enhancing efficiency, productivity, and offering novel solutions to complex challenges. Foundation models are promising in automating complex tasks, though challenges remain in scientific reliability, bias, interpretability, and workflow integration.
  arXiv  Detail & Related papers  (2024-10-26T15:42:50Z)
• Collaborative AI in Sentiment Analysis: System Architecture, Data Prediction and Deployment Strategies [3.3374611485861116]
  Large language model (LLM) based artificial intelligence technologies have been a game-changer, particularly in sentiment analysis. However, integrating diverse AI models for processing complex multimodal data and the associated high costs of feature extraction presents significant challenges. This study introduces a collaborative AI framework designed to efficiently distribute and resolve tasks across various AI systems.
  arXiv  Detail & Related papers  (2024-10-17T06:14:34Z)
• Proactive Agent: Shifting LLM Agents from Reactive Responses to Active Assistance [95.03771007780976]
  We tackle the challenge of developing proactive agents capable of anticipating and initiating tasks without explicit human instructions. First, we collect real-world human activities to generate proactive task predictions. These predictions are labeled by human annotators as either accepted or rejected. The labeled data is used to train a reward model that simulates human judgment.
  arXiv  Detail & Related papers  (2024-10-16T08:24:09Z)
• Measuring Human Contribution in AI-Assisted Content Generation [68.03658922067487]
  This study raises the research question of measuring human contribution in AI-assisted content generation. By calculating mutual information between human input and AI-assisted output relative to self-information of AI-assisted output, we quantify the proportional information contribution of humans in content generation.
  arXiv  Detail & Related papers  (2024-08-27T05:56:04Z)
• Generative AI for Health Technology Assessment: Opportunities, Challenges, and Policy Considerations [12.73011921253]
  This review introduces the transformative potential of generative Artificial Intelligence (AI) and foundation models, including large language models (LLMs), for health technology assessment (HTA) We explore their applications in four critical areas, synthesis evidence, evidence generation, clinical trials and economic modeling. Despite their promise, these technologies, while rapidly improving, are still nascent and continued careful evaluation in their applications to HTA is required.
  arXiv  Detail & Related papers  (2024-07-09T09:25:27Z)
• Augmenting the Author: Exploring the Potential of AI Collaboration in Academic Writing [25.572926673827165]
  This case study highlights the importance of prompt design, output analysis, and recognizing the AI's limitations to ensure responsible and effective AI integration in scholarly work. The paper contributes to the field of Human-Computer Interaction by exploring effective prompt strategies and providing a comparative analysis of Gen AI models.
  arXiv  Detail & Related papers  (2024-04-23T19:06:39Z)
• Generative AI Agent for Next-Generation MIMO Design: Fundamentals, Challenges, and Vision [76.4345564864002]
  Next-generation multiple input multiple output (MIMO) is expected to be intelligent and scalable. We propose the concept of the generative AI agent, which is capable of generating tailored and specialized contents. We present two compelling case studies that demonstrate the effectiveness of leveraging the generative AI agent for performance analysis.
  arXiv  Detail & Related papers  (2024-04-13T02:39:36Z)
• An Autonomous Large Language Model Agent for Chemical Literature Data Mining [60.85177362167166]
  We introduce an end-to-end AI agent framework capable of high-fidelity extraction from extensive chemical literature. Our framework's efficacy is evaluated using accuracy, recall, and F1 score of reaction condition data.
  arXiv  Detail & Related papers  (2024-02-20T13:21:46Z)
• Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolution [92.84441068115517]
  Investigate-Consolidate-Exploit (ICE) is a novel strategy for enhancing the adaptability and flexibility of AI agents. ICE promotes the transfer of knowledge between tasks for genuine self-evolution. Our experiments on the XAgent framework demonstrate ICE's effectiveness, reducing API calls by as much as 80%.
  arXiv  Detail & Related papers  (2024-01-25T07:47:49Z)
• Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-based Recruitment [66.91538273487379]
  There is a certain consensus about the need to develop AI applications with a Human-Centric approach. Human-Centric Machine Learning needs to be developed based on four main requirements: (i) utility and social good; (ii) privacy and data ownership; (iii) transparency and accountability; and (iv) fairness in AI-driven decision-making processes. We study how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data.
  arXiv  Detail & Related papers  (2023-02-13T16:44:44Z)
• An Explainable-AI approach for Diagnosis of COVID-19 using MALDI-ToF Mass Spectrometry [0.9250974571641537]
  Severe acute respiratory syndrome coronavirus type-2 (SARS-CoV-2) caused a global pandemic and immensely affected the global economy. Recently, multiple alternative platforms for testing coronavirus disease 2019 (COVID-19) have been published that show high agreement with current gold standard real-time polymerase chain reaction (RT-PCR) results. These new methods do away with nasopharyngeal (NP) swabs, eliminate the need for complicated reagents, and reduce the burden on RT-PCR test reagent supply. In the present work, we have designed an artificial intelligence-based (AI) testing method to provide confidence in the results.
  arXiv  Detail & Related papers  (2021-09-28T23:29:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.