Journal article
LLM-Generated Multiple Choice Practice Quizzes for Pre-Clinical Medical Students; Prevalence of Item Writing Flaws
Advances in physiology education, v 49(3)
14 Jun 2025
PMID: 40516963
Abstract
Multiple choice questions (MCQs) are frequently used in medical education for assessment. Automated generation of MCQs in board-exam format could potentially save significant effort for faculty and generate a wider set of practice materials for student use. The goal of this study was to explore the feasibility of using ChatGPT by OpenAI to generate USMLE/COMLEX-USA-style practice quiz items as study aids. Researchers gave second year medical students studying renal physiology access to a set of practice quizzes with ChatGPT generated questions. The exam items generated were evaluated by independent experts for quality and adherence to NBME/NBOME guidelines. Forty-nine percent of questions contained item writing flaws, and 22% contained factual or conceptual errors. However, 59/65 (91%) were categorized as a reasonable starting point for revision. These results demonstrate the feasibility of large language model (LLM)-generated practice questions in medical education, but only when supervised by a subject matter expert with training in exam item writing.
Metrics
31 Record Views
Details
- Title
- LLM-Generated Multiple Choice Practice Quizzes for Pre-Clinical Medical Students; Prevalence of Item Writing Flaws
- Creators
- Troy Camarata - Southwest Baptist UniversityLise McCoy - New York Institute of Technology College of Osteopathic MedicineRobert L Rosenberg - Drexel UniversityKelsey R Temprine Grellinger - Western Michigan UniversityKylie Brettschneider - New York Institute of Technology College of Osteopathic MedicineJonathan Berman - New York Institute of Technology College of Osteopathic Medicine
- Publication Details
- Advances in physiology education, v 49(3)
- Publisher
- American Physiological Society
- Number of pages
- 6
- Resource Type
- Journal article
- Language
- English
- Academic Unit
- Pharmacology and Physiology
- Web of Science ID
- WOS:001566756600001
- Scopus ID
- 2-s2.0-105009935071
- Other Identifier
- 991022059818004721
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Collaboration types
- Domestic collaboration
- Web of Science research areas
- Education, Scientific Disciplines
- Physiology