Performance of a large language model-Artificial Intelligence based chatbot for counseling patients with sexually transmitted infections and genital diseases

• URL: http://arxiv.org/abs/2412.12166v1
• Date: Wed, 11 Dec 2024 20:36:32 GMT
• Title: Performance of a large language model-Artificial Intelligence based chatbot for counseling patients with sexually transmitted infections and genital diseases
• Authors: Nikhil Mehta, Sithira Ambepitiya, Thanveer Ahamad, Dinuka Wijesundara, Yudara Kularathne,
• Abstract summary: Otiz is an AI-based platform designed specifically for STI detection and counseling. Four STIs (anogenital warts, herpes, syphilis, urethritis/cervicitis) were evaluated using prompts mimicking patient language. Otiz scored highly on diagnostic accuracy (4.14.7), overall accuracy (4.34.6), correctness of information (5.0), comprehensibility (4.2-4.4), and empathy (4.5-4.3.6)
• Score: 4.910821423749911
• License:
• Abstract: Introduction: Global burden of sexually transmitted infections (STIs) is rising out of proportion to specialists. Current chatbots like ChatGPT are not tailored for handling STI-related concerns out of the box. We developed Otiz, an Artificial Intelligence-based (AI-based) chatbot platform designed specifically for STI detection and counseling, and assessed its performance. Methods: Otiz employs a multi-agent system architecture based on GPT4-0613, leveraging large language model (LLM) and Deterministic Finite Automaton principles to provide contextually relevant, medically accurate, and empathetic responses. Its components include modules for general STI information, emotional recognition, Acute Stress Disorder detection, and psychotherapy. A question suggestion agent operates in parallel. Four STIs (anogenital warts, herpes, syphilis, urethritis/cervicitis) and 2 non-STIs (candidiasis, penile cancer) were evaluated using prompts mimicking patient language. Each prompt was independently graded by two venereologists conversing with Otiz as patient actors on 6 criteria using Numerical Rating Scale ranging from 0 (poor) to 5 (excellent). Results: Twenty-three venereologists did 60 evaluations of 30 prompts. Across STIs, Otiz scored highly on diagnostic accuracy (4.1-4.7), overall accuracy (4.3-4.6), correctness of information (5.0), comprehensibility (4.2-4.4), and empathy (4.5-4.8). However, relevance scores were lower (2.9-3.6), suggesting some redundancy. Diagnostic scores for non-STIs were lower (p=0.038). Inter-observer agreement was strong, with differences greater than 1 point occurring in only 12.7% of paired evaluations. Conclusions: AI conversational agents like Otiz can provide accurate, correct, discrete, non-judgmental, readily accessible and easily understandable STI-related information in an empathetic manner, and can alleviate the burden on healthcare systems.

Related papers

• A Scalable Approach to Benchmarking the In-Conversation Differential Diagnostic Accuracy of a Health AI [0.0]
  This study introduces a scalable benchmarking methodology for assessing health AI systems. Our methodology employs 400 validated clinical vignettes across 14 medical specialties, using AI-powered patient actors to simulate realistic clinical interactions. August achieved a top-one diagnostic accuracy of 81.8% (327/400 cases) and a top-two accuracy of 85.0% (340/400 cases), significantly outperforming traditional symptom checkers.
  arXiv  Detail & Related papers  (2024-12-17T05:02:33Z)
• Improving Clinical Documentation with AI: A Comparative Study of Sporo AI Scribe and GPT-4o mini [0.0]
  Sporo Health's AI scribe was evaluated against OpenAI's GPT-4o Mini. Results show that Sporo AI consistently outperformed GPT-4o Mini, achieving higher recall, precision, and overall F1 scores.
  arXiv  Detail & Related papers  (2024-10-20T22:48:40Z)
• A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor? [33.70022886795487]
  OpenAI's o1 stands out as the first model with a chain-of-thought technique using reinforcement learning strategies. This report provides a comprehensive exploration of o1 on different medical scenarios, examining 3 key aspects: understanding, reasoning, and multilinguality.
  arXiv  Detail & Related papers  (2024-09-23T17:59:43Z)
• GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI [67.09501109871351]
  Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals. GMAI-MMBench is the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format.
  arXiv  Detail & Related papers  (2024-08-06T17:59:21Z)
• Evaluating the Application of ChatGPT in Outpatient Triage Guidance: A Comparative Study [11.37622565068147]
  The integration of Artificial Intelligence in healthcare presents a transformative potential for enhancing operational efficiency and health outcomes. Large Language Models (LLMs), such as ChatGPT, have shown their capabilities in supporting medical decision-making. This study specifically aims to evaluate the consistency of responses provided by ChatGPT in outpatient guidance.
  arXiv  Detail & Related papers  (2024-04-27T04:12:02Z)
• Towards Conversational Diagnostic AI [32.84876349808714]
  We introduce AMIE (Articulate Medical Intelligence Explorer), a Large Language Model (LLM) based AI system optimized for diagnostic dialogue. AMIE uses a self-play based simulated environment with automated feedback mechanisms for scaling learning across diverse disease conditions. AMIE demonstrated greater diagnostic accuracy and superior performance on 28 of 32 axes according to specialist physicians and 24 of 26 axes according to patient actors.
  arXiv  Detail & Related papers  (2024-01-11T04:25:06Z)
• Automatically measuring speech fluency in people with aphasia: first achievements using read-speech data [55.84746218227712]
  This study aims at assessing the relevance of a signalprocessingalgorithm, initially developed in the field of language acquisition, for the automatic measurement of speech fluency.
  arXiv  Detail & Related papers  (2023-08-09T07:51:40Z)
• Exploiting prompt learning with pre-trained language models for Alzheimer's Disease detection [70.86672569101536]
  Early diagnosis of Alzheimer's disease (AD) is crucial in facilitating preventive care and to delay further progression. This paper investigates the use of prompt-based fine-tuning of PLMs that consistently uses AD classification errors as the training objective function.
  arXiv  Detail & Related papers  (2022-10-29T09:18:41Z)
• Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation [56.25869366777579]
  In recent years, machine learning models have rapidly become better at generating clinical consultation notes. We present an extensive human evaluation study where 5 clinicians listen to 57 mock consultations, write their own notes, post-edit a number of automatically generated notes, and extract all the errors. We find that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore.
  arXiv  Detail & Related papers  (2022-04-01T14:04:16Z)
• Advancing COVID-19 Diagnosis with Privacy-Preserving Collaboration in Artificial Intelligence [79.038671794961]
  We launch the Unified CT-COVID AI Diagnostic Initiative (UCADI), where the AI model can be distributedly trained and independently executed at each host institution. Our study is based on 9,573 chest computed tomography scans (CTs) from 3,336 patients collected from 23 hospitals located in China and the UK.
  arXiv  Detail & Related papers  (2021-11-18T00:43:41Z)
• NUVA: A Naming Utterance Verifier for Aphasia Treatment [49.114436579008476]
  Assessment of speech performance using picture naming tasks is a key method for both diagnosis and monitoring of responses to treatment interventions by people with aphasia (PWA) Here we present NUVA, an utterance verification system incorporating a deep learning element that classifies 'correct' versus'incorrect' naming attempts from aphasic stroke patients. When tested on eight native British-English speaking PWA the system's performance accuracy ranged between 83.6% to 93.6%, with a 10-fold cross-validation mean of 89.5%.
  arXiv  Detail & Related papers  (2021-02-10T13:00:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.