Files and links (1)
Preprint (Author's original)arXiv.org - Non-exclusive license to distribute, Open
Journal article
An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers
27 Jul 2021
Abstract
With the advent of off-the-shelf intelligent home products and broader
internet adoption, researchers increasingly explore smart computing
applications that provide easier access to health and wellness resources.
AI-based systems like chatbots have the potential to provide services that
could provide mental health support. However, existing therapy chatbots are
often retrieval-based, requiring users to respond with a constrained set of
answers, which may not be appropriate given that such pre-determined inquiries
may not reflect each patient's unique circumstances. Generative-based
approaches, such as the OpenAI GPT models, could allow for more dynamic
conversations in therapy chatbot contexts than previous approaches. To
investigate the generative-based model's potential in therapy chatbot contexts,
we built a chatbot using the GPT-2 model. We fine-tuned it with 306 therapy
session transcripts between family caregivers of individuals with dementia and
therapists conducting Problem Solving Therapy. We then evaluated the model's
pre-trained and the fine-tuned model in terms of basic qualities using three
meta-information measurements: the proportion of non-word outputs, the length
of response, and sentiment components. Results showed that: (1) the fine-tuned
model created more non-word outputs than the pre-trained model; (2) the
fine-tuned model generated outputs whose length was more similar to that of the
therapists compared to the pre-trained model; (3) both the pre-trained model
and fine-tuned model were likely to generate more negative and fewer positive
outputs than the therapists. We discuss potential reasons for the problem, the
implications, and solutions for developing therapy chatbots and call for
investigations of the AI-based system application.
Metrics
10 Record Views
Details
- Title
- An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregivers
- Creators
- Lu WangMunif Ishad MujibJake WilliamsGeorge DemirisJina Huh-Yoo
- Resource Type
- Journal article
- Language
- English
- Academic Unit
- Information Science (Informatics)
- Other Identifier
- 991019297629604721