Testing the viability of large language model artificial intelligence as a form of academic support for non-U.S. students preparing for the critical analysis and reasoning skills section of the Medical College Admission Test

Ilan Friedman

doi:10.17918/00010993

This study explores the potential of large language model artificial intelligence as a form of academic support for English Language Learner (ELL) students, who are foreign to the United States, who are preparing to take the Critical Analysis and Reasoning Skills (CARS) section of the Medical College Admission Test (MCAT); this section generally sees significantly lower scores than others for this population, likely due to differences in linguistics and culture between these students and the test writers, as the questions generally involve passages which reflect culture based in the United States, and not necessarily due to a lack of the critical thinking skills for which the section is supposed to test. The researcher prompted ChatGPT to generate 540 CARS sample questions, answers, and explanations about why answers were correct. The researcher prompted ChatGPT to cater 20 percent of explanations to students from India, 20 percent to students from Pakistan, 20 percent to students from Lebanon, 20 percent to students from Nepal, and 20 percent to students from Jordan. These are the nations, without English as a native language, from which the most students attend United States medical schools. Following qualitative analysis, found consistencies and themes were compared to best practices in ELL education, U.S. medical school critical thinking learning objectives, and cultural sensitivity for college students, based on conclusions drawn by Phinney's Ethnic Identity Development Model and Molinsky's Science of Diagnosing Cultural Differences. There were a few primary conclusions drawn by the researcher. To start, ChatGPT was sufficient as a form of academic support when focusing specifically on the test itself; the parts of explanations which discussed why an answer was correct or not performed well. In doing so, it consistently connected valuable CARS tips to the rest of the explanation; these tips included the value of recognizing nuance, taking on the author's perspective rather than one's own, and noting when an author is making an argument versus exploring a debate. There was marked room for improvement when ChatGPT attempted to connect these explanations to the students. Often it relied on cultural overgeneralizations, being incorrect, and a writing tone which did not work to cater to anybody from any specific nation. While ChatGPT can be a strong general tutor, it is not yet at a point where it can be one customized to a student's particular culture. With this in mind, it, in this study, had foregone main tenets of successful ELL education, could not take into account cultural sensitivity, but did successfully teach about U.S. medical school critical thinking learning objectives. In the future, with more data intake, the researcher anticipates an improvement. The researcher suggests that in practice, this be used by educators, who then fill gaps in connecting material to students, and with regard to future research, the researcher suggests the changes in scores of this student population for this test section moving forward, relative to their use of ChatGPT, ChatGPT with a teacher or tutor, with just a teacher or tutor, and independently.

Testing the viability of large language model artificial intelligence as a form of academic support for non-U.S. students preparing for the critical analysis and reasoning skills section of the Medical College Admission Test

Files and links (1)

Abstract

Metrics

Details

Testing the viability of large language model artificial intelligence as a form of academic support for non-U.S. students preparing for the critical analysis and reasoning skills section of the Medical College Admission Test

Files and links (1)

Abstract

Metrics

Details

Drexel University Social media