Automated Survey Collection with LLM-based Conversational Agents

• URL: http://arxiv.org/abs/2504.02891v1
• Date: Wed, 02 Apr 2025 18:10:19 GMT
• Title: Automated Survey Collection with LLM-based Conversational Agents
• Authors: Kurmanbek Kaiyrbekov, Nicholas J Dobbins, Sean D Mooney,
• Abstract summary: Traditional phone-based surveys are among the most accessible and widely used methods to collect biomedical and healthcare data.<n>We propose an end-to-end survey collection framework driven by conversational Large Language Models (LLMs)<n>Our framework consists of a researcher responsible for designing the survey and recruiting participants, a conversational phone agent powered by an LLM that calls participants and administers the survey, a second LLM that analyzes the conversation transcripts generated during the surveys, and a database for storing and organizing the results.
• Score: 1.9915268274949234
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Objective: Traditional phone-based surveys are among the most accessible and widely used methods to collect biomedical and healthcare data, however, they are often costly, labor intensive, and difficult to scale effectively. To overcome these limitations, we propose an end-to-end survey collection framework driven by conversational Large Language Models (LLMs). Materials and Methods: Our framework consists of a researcher responsible for designing the survey and recruiting participants, a conversational phone agent powered by an LLM that calls participants and administers the survey, a second LLM (GPT-4o) that analyzes the conversation transcripts generated during the surveys, and a database for storing and organizing the results. To test our framework, we recruited 8 participants consisting of 5 native and 3 non-native english speakers and administered 40 surveys. We evaluated the correctness of LLM-generated conversation transcripts, accuracy of survey responses inferred by GPT-4o and overall participant experience. Results: Survey responses were successfully extracted by GPT-4o from conversation transcripts with an average accuracy of 98% despite transcripts exhibiting an average per-line word error rate of 7.7%. While participants noted occasional errors made by the conversational LLM agent, they reported that the agent effectively conveyed the purpose of the survey, demonstrated good comprehension, and maintained an engaging interaction. Conclusions: Our study highlights the potential of LLM agents in conducting and analyzing phone surveys for healthcare applications. By reducing the workload on human interviewers and offering a scalable solution, this approach paves the way for real-world, end-to-end AI-powered phone survey collection systems.

Related papers

• Highlighting Case Studies in LLM Literature Review of Interdisciplinary System Science [0.18416014644193066]
  Large Language Models (LLMs) were used to assist four Commonwealth Scientific and Industrial Research Organisation (CSIRO) researchers. We evaluate the performance of LLMs for systematic literature reviews.
  arXiv  Detail & Related papers  (2025-03-16T05:52:18Z)
• Telephone Surveys Meet Conversational AI: Evaluating a LLM-Based Telephone Survey System at Scale [0.0]
  This presents an AI-driven telephone survey system integrating text-to-speech (TTS), a large language model (LLM), and speech-to-speech (STT)<n>We tested the system across two populations, a pilot study in the United States (n = 75) and a large-scale deployment in Peru (n = 2,739)<n>Our findings demonstrate that while the AI system's probing for qualitative depth was more limited than human interviewers, overall data quality approached human-led standards for structured items.
  arXiv  Detail & Related papers  (2025-02-27T14:31:42Z)
• GuideLLM: Exploring LLM-Guided Conversation with Applications in Autobiography Interviewing [73.8469700907927]
  Large Language Models (LLMs) succeed in human-guided conversations such as instruction following and question answering.<n>In this study, we first characterize LLM-guided conversation into three fundamental components: Goal Navigation; (ii) Context Management; (iii) Empathetic Engagement.<n>We compare GuideLLM with 6 state-of-the-art LLMs such as GPT-4o and Llama-3-70b-Instruct, from the perspective of interviewing quality, and autobiography generation quality.
  arXiv  Detail & Related papers  (2025-02-10T14:11:32Z)
• NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews [65.35458530702442]
  We focus on journalistic interviews, a domain rich in grounding communication and abundant in data. We curate a dataset of 40,000 two-person informational interviews from NPR and CNN. LLMs are significantly less likely than human interviewers to use acknowledgements and to pivot to higher-level questions.
  arXiv  Detail & Related papers  (2024-11-21T01:37:38Z)
• FactFinders at CheckThat! 2024: Refining Check-worthy Statement Detection with LLMs through Data Pruning [43.82613670331329]
  This study investigates the application of open-source language models to identify check-worthy statements from political transcriptions. We propose a two-step data pruning approach to automatically identify high-quality training data instances for effective learning. Our team ranked first in the check-worthiness estimation task in the English language.
  arXiv  Detail & Related papers  (2024-06-26T12:31:31Z)
• Exploring the Factual Consistency in Dialogue Comprehension of Large Language Models [51.75805497456226]
  This work focuses on the factual consistency issue with the help of the dialogue summarization task. Our evaluation shows that, on average, 26.8% of the summaries generated by LLMs contain factual inconsistency. To stimulate and enhance the dialogue comprehension ability of LLMs, we propose a fine-tuning paradigm with auto-constructed multi-task data.
  arXiv  Detail & Related papers  (2023-11-13T09:32:12Z)
• MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records [60.35217378132709]
  Large language models (LLMs) can follow natural language instructions with human-level fluency. evaluating LLMs on realistic text generation tasks for healthcare remains challenging. We introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data.
  arXiv  Detail & Related papers  (2023-08-27T12:24:39Z)
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback [127.75419038610455]
  Large language models (LLMs) are able to generate human-like, fluent responses for many downstream tasks. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules.
  arXiv  Detail & Related papers  (2023-02-24T18:48:43Z)
• Partner Matters! An Empirical Study on Fusing Personas for Personalized Response Selection in Retrieval-Based Chatbots [51.091235903442715]
  This paper makes an attempt to explore the impact of utilizing personas that describe either self or partner speakers on the task of response selection. Four persona fusion strategies are designed, which assume personas interact with contexts or responses in different ways. Empirical studies on the Persona-Chat dataset show that the partner personas can improve the accuracy of response selection.
  arXiv  Detail & Related papers  (2021-05-19T10:32:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.