Journal article
Class in Session: Analysis of GPT-4-created Plastic Surgery In-service Examination Questions
Plastic and reconstructive surgery. Global open, v 12(9), 6185
01 Sep 2024
PMID: 39301304
Featured in Collection : UN Sustainable Development Goals @ Drexel
Abstract
Background:The Plastic Surgery In-Service Training Examination (PSITE) remains a critical milestone in residency training. Successful preparation requires extensive studying during an individual's residency. This study focuses on the capacity of Generative Pre-trained Transformer 4 (GPT-4) to generate PSITE practice questions. Methods:GPT-4 was prompted to generate multiple choice questions for each PSITE section and provide answer choices with detailed rationale. Question composition via readability metrics were analyzed, along with quality. Descriptive statistics compared GPT-4 and the 2022 PSITE. Results:The overall median Flesch-Kincaid reading ease for GPT-4-generated questions was 43.90 (versus 50.35 PSITE, P = 0.036). GPT-4 provided questions that contained significantly fewer mean sentences (1 versus 4), words (16 versus 56), and percentage of complex words (3 versus 13) than 2022 PSITE questions (P < 0.001). When evaluating GPT-4 generated questions for each examination section, the highest median Flesch-Kincaid reading ease was on the core surgical principles section (median: 63.30, interquartile range [54.45-68.28]) and the lowest was on the craniomaxillofacial section (median: 36.25, interquartile range [12.57-58.40]). Most readability metrics were higher for the 2022 PSITE compared with GPT-4 generated questions. Overall question quality was poor for the chatbot. Conclusions:Our study found that GPT-4 can be adapted to generate practice questions for the 2022 PSITE, but its questions are of poor quality. The program can offer general explanations for both the correct and incorrect answer options but was observed to generate false information and poor-quality explanations. Although trainees should navigate with caution as the technology develops, GPT-4 has the potential to serve as an effective educational adjunct under the supervision of trained plastic surgeons.
Metrics
1 Record Views
Details
- Title
- Class in Session: Analysis of GPT-4-created Plastic Surgery In-service Examination Questions
- Creators
- Daniel Najafali - University of Illinois Urbana-ChampaignLogan G. Galbraith - Northeast Ohio Medical UniversityJustin M. Camacho - Drexel UniversityVictoria Stoffel - Drexel UniversityIsabel Herzog - Rutgers, The State University of New JerseyCivanni Moss - Temple UniversityStephanie L. Taiberg - Rosalind Franklin University of Medicine and ScienceLeonard Knoedler (Corresponding Author) - Massachusetts General Hospital
- Publication Details
- Plastic and reconstructive surgery. Global open, v 12(9), 6185
- Publisher
- Lippincott Williams & Wilkins
- Number of pages
- 6
- Resource Type
- Journal article
- Language
- English
- Academic Unit
- College of Medicine
- Web of Science ID
- WOS:001371418000001
- Scopus ID
- 2-s2.0-85205027423
- Other Identifier
- 991022202098904721
UN Sustainable Development Goals (SDGs)
This publication has contributed to the advancement of the following goals:
Source: SDGs in the Output
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Collaboration types
- Domestic collaboration
- International collaboration
- Web of Science research areas
- Surgery