An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers

• URL: http://arxiv.org/abs/2107.13115v1
• Date: Wed, 28 Jul 2021 01:01:08 GMT
• Title: An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers
• Authors: Lu Wang, Munif Ishad Mujib, Jake Williams, George Demiris, Jina Huh-Yoo
• Abstract summary: Generative-based approaches, such as the OpenAI GPT models, could allow for more dynamic conversations in therapy contexts. We built a chatbots using the GPT-2 model and fine-tuned it with 306 therapy session transcripts between family caregivers of individuals with dementia and therapists conducting Problem Solving Therapy. Results showed that the fine-tuned model created more non-word outputs than the pre-trained model.
• Score: 5.2116528363639985
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the advent of off-the-shelf intelligent home products and broader internet adoption, researchers increasingly explore smart computing applications that provide easier access to health and wellness resources. AI-based systems like chatbots have the potential to provide services that could provide mental health support. However, existing therapy chatbots are often retrieval-based, requiring users to respond with a constrained set of answers, which may not be appropriate given that such pre-determined inquiries may not reflect each patient's unique circumstances. Generative-based approaches, such as the OpenAI GPT models, could allow for more dynamic conversations in therapy chatbot contexts than previous approaches. To investigate the generative-based model's potential in therapy chatbot contexts, we built a chatbot using the GPT-2 model. We fine-tuned it with 306 therapy session transcripts between family caregivers of individuals with dementia and therapists conducting Problem Solving Therapy. We then evaluated the model's pre-trained and the fine-tuned model in terms of basic qualities using three meta-information measurements: the proportion of non-word outputs, the length of response, and sentiment components. Results showed that: (1) the fine-tuned model created more non-word outputs than the pre-trained model; (2) the fine-tuned model generated outputs whose length was more similar to that of the therapists compared to the pre-trained model; (3) both the pre-trained model and fine-tuned model were likely to generate more negative and fewer positive outputs than the therapists. We discuss potential reasons for the problem, the implications, and solutions for developing therapy chatbots and call for investigations of the AI-based system application.

Related papers

• Conversational Topic Recommendation in Counseling and Psychotherapy with Decision Transformer and Large Language Models [17.236038165057817]
  We leverage a decision transformer architecture for topic recommendation in counseling conversations. The architecture is utilized for offline reinforcement learning. We propose a novel system of utilizing our model's output as synthetic labels for fine-tuning a large language model.
  arXiv  Detail & Related papers  (2024-05-08T13:55:25Z)
• Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback [20.73076574974894]
  We propose an approach called preference learning from process feedback. PLPF integrates the doctor's diagnostic logic into LLMs. We show that PLPF enhances the diagnostic accuracy of the baseline model in medical conversations by 17.6%.
  arXiv  Detail & Related papers  (2024-01-11T06:42:45Z)
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models [115.501751261878]
  Fine-tuning language models(LMs) on human-generated data remains a prevalent practice. We investigate whether we can go beyond human data on tasks where we have access to scalar feedback. We find that ReST$EM$ scales favorably with model size and significantly surpasses fine-tuning only on human data.
  arXiv  Detail & Related papers  (2023-12-11T18:17:43Z)
• Development and Evaluation of Three Chatbots for Postpartum Mood and Anxiety Disorders [31.018188794627378]
  We develop three chatbots to provide context-specific empathetic support to postpartum caregivers. We present and evaluate the performance of our chatbots using both machine-based metrics and human-based questionnaires. We conclude by discussing practical benefits of rule-based vs. generative models for supporting individuals with mental health challenges.
  arXiv  Detail & Related papers  (2023-08-14T18:52:03Z)
• BiomedGPT: A Unified and Generalist Biomedical Generative Pre-trained Transformer for Vision, Language, and Multimodal Tasks [68.39821375903591]
  Generalist AI holds the potential to address limitations due to its versatility in interpreting different data types. Here, we propose BiomedGPT, the first open-source and lightweight vision-language foundation model.
  arXiv  Detail & Related papers  (2023-05-26T17:14:43Z)
• Safe AI for health and beyond -- Monitoring to transform a health service [51.8524501805308]
  We will assess the infrastructure required to monitor the outputs of a machine learning algorithm. We will present two scenarios with examples of monitoring and updates of models.
  arXiv  Detail & Related papers  (2023-03-02T17:27:45Z)
• Transformers for prompt-level EMA non-response prediction [62.41658786277712]
  Ecological Momentary Assessments (EMAs) are an important psychological data source for measuring cognitive states, affect, behavior, and environmental factors. Non-response, in which participants fail to respond to EMA prompts, is an endemic problem. The ability to accurately predict non-response could be utilized to improve EMA delivery and develop compliance interventions.
  arXiv  Detail & Related papers  (2021-11-01T18:38:47Z)
• Generative Adversarial Imitation Learning for Empathy-based AI [0.0]
  Generative adversarial imitation learning (GAIL) is a model-free algorithm that has been shown to provide strong results in imitating complex behaviors in high-dimensional environments. In this paper, we utilize the GAIL model for text generation to develop empathy-based context-aware conversational AI.
  arXiv  Detail & Related papers  (2021-05-27T17:37:37Z)
• Towards Automated Psychotherapy via Language Modeling [0.0]
  The model was trained upon a mix of the Cornell Movie Dialogue Corpus for language understanding and an open-source, anonymized, and public licensed psychotherapeutic dataset. The model achieved statistically significant performance in published, standardized qualitative benchmarks against human-written validation data. Although the model cannot replace the work of psychotherapists entirely, its ability to synthesize human-appearing utterances for the majority of the test set serves as a promising step towards communizing and easing stigma at the psychotherapeutic point-of-care.
  arXiv  Detail & Related papers  (2021-04-05T01:53:39Z)
• Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach [84.02388020258141]
  We propose a new framework named ENIGMA for estimating human evaluation scores based on off-policy evaluation in reinforcement learning. ENIGMA only requires a handful of pre-collected experience data, and therefore does not involve human interaction with the target policy during the evaluation. Our experiments show that ENIGMA significantly outperforms existing methods in terms of correlation with human evaluation scores.
  arXiv  Detail & Related papers  (2021-02-20T03:29:20Z)
• Decoding Methods for Neural Narrative Generation [74.37264021226308]
  Narrative generation is an open-ended NLP task in which a model generates a story given a prompt. We apply and evaluate advances in decoding methods for neural response generation to neural narrative generation.
  arXiv  Detail & Related papers  (2020-10-14T19:32:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.