Journal article
Evaluating Generative Pretrained Transformer (GPT) models for suicide risk assessment in synthetic patient journal entries
BMC psychiatry, v 25(1), 753
01 Aug 2025
PMID: 40750858
Featured in Collection : UN Sustainable Development Goals @ Drexel
Abstract
Over 700,000 individuals die by suicide globally each year, with rapid progression from suicidal ideation (SI) to attempt often precluding opportunities for intervention. Digital behavioral health (DBH) platforms offer novel means of collecting SI indicators outside the clinic, but the actionable utility of these data may be limited by clinician-dependent workflows such as reviewing patients' journaling exercises for signs of SI. Large language models (LLMs) provide a methodology to streamline this task by rapidly risk-stratifying text based on the presence and severity of SI; however, this application has yet to be reliably evaluated. To test this approach, we first generated and validated a corpus of 125 synthetic journal responses to prompts from a real-world DBH platform. The responses varied on the presence and severity of suicidal ideation, readability, length, use of emojis, and other common language features, allowing for over 1 trillion feature permutations. Next, five collaborating behavioral health experts worked independently to stratify these responses as no-, low-, moderate-, or high-risk SI. Finally, we risk-stratified the responses using several tailored implementations of OpenAI's Generative Pretrained Transformer (GPT) models and compared the results to those of our raters. Using clinician consensus as "ground truth," our ensemble LLM performed significantly above chance (30.38%) in exact risk-assessment agreement (65.60%; χ2 = 86.58). The ensemble model also aligned with 92% of clinicians' "do/do not intervene" decisions (Cohen's Kappa = 0.84) and achieved 94% sensitivity and 91% specificity in that task. Additional results of precision-recall, time-to-decision, and cost analyses are reported. While further testing and exploration of ethical considerations remain critical, our results offer preliminary evidence that LLM-powered risk stratification can serve as a powerful and cost-effective tool to enhance suicide prevention frameworks.
Metrics
5 Record Views
Details
- Title
- Evaluating Generative Pretrained Transformer (GPT) models for suicide risk assessment in synthetic patient journal entries
- Creators
- Dan Holley - NeuroFlow (United States_Philadelphia)Brian Daly - Drexel UniversityBriana Beverly - NeuroFlow (United States_Philadelphia)Blaken Wamsley - NeuroFlow (United States_Philadelphia)Amanda Brooks - NeuroFlow (United States_Philadelphia)Tom Zaubler - NeuroFlow (United States_Philadelphia)
- Publication Details
- BMC psychiatry, v 25(1), 753
- Publisher
- BMC
- Number of pages
- 14
- Grant note
- NeuroFlow
The study team thanks Drs. Petra Kottsieper and Mark Sander for data provided, which were crucial to our analyses. We also thank Brie Bernhardt and Faith Best for clinical insights during our prompt-engineering efforts. Finally, we thank Matthew Miclette for his insights, feedback, and constant enthusiasm for AI as a transformative technology for healthcare.
- Resource Type
- Journal article
- Language
- English
- Academic Unit
- Psychological and Brain Sciences (Psychology)
- Web of Science ID
- WOS:001542631300003
- Scopus ID
- 2-s2.0-105012392931
- Other Identifier
- 991022065638304721
UN Sustainable Development Goals (SDGs)
This publication has contributed to the advancement of the following goals:
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Web of Science research areas
- Psychiatry