Journal article
Evaluating ChatGPT-4 in medical education: an assessment of subject exam performance reveals limitations in clinical curriculum support for students
Discover Artificial Intelligence, v 4(1), pp 38-5
01 Dec 2024
Abstract
This study evaluates the proficiency of ChatGPT-4 across various medical specialties and assesses its potential as a study tool for medical students preparing for the United States Medical Licensing Examination (USMLE) Step 2 and related clinical subject exams. ChatGPT-4 answered board-level questions with 89% accuracy, but showcased significant discrepancies in performance across specialties. Although it excelled in psychiatry, neurology, and obstetrics and gynecology, it underperformed in pediatrics, emergency medicine, and family medicine. These variations may be potentially attributed to the depth and recency of training data as well as the scope of the specialties assessed. Specialties with significant interdisciplinary overlap had lower performance, suggesting complex clinical scenarios pose a challenge to the AI. In terms of the future, the overall efficacy of ChatGPT-4 indicates a promising supplemental role in medical education, but performance inconsistencies across specialties in the current version lead us to recommend that medical students use AI with caution.
Metrics
10 Record Views
4 citations in Scopus
Details
- Title
- Evaluating ChatGPT-4 in medical education: an assessment of subject exam performance reveals limitations in clinical curriculum support for students
- Creators
- Brendan P. Mackey - Drexel UniversityRazmig Garabet - Drexel UniversityLaura Maule - Drexel UniversityAbay Tadesse - Drexel UniversityJames Cross - Drexel UniversityMichael Weingarten - Drexel University
- Publication Details
- Discover Artificial Intelligence, v 4(1), pp 38-5
- Publisher
- Springer International Publishing
- Resource Type
- Journal article
- Language
- English
- Academic Unit
- MD (Doctor of Medicine) Program; Surgery
- Scopus ID
- 2-s2.0-85193540066
- Other Identifier
- 991021960797904721