Journal article
Chatbot responses suggest that hypothetical biology questions are harder than realistic ones
Journal of microbiology & biology education, v 24(3)
01 Dec 2023
PMID: 38107990
Featured in Collection : UN Sustainable Development Goals @ Drexel
Abstract
The biology education literature includes compelling assertions that unfamiliar problems are especially useful for revealing students’ true understanding of biology. However, there is only limited evidence that such novel problems have different cognitive requirements than more familiar problems. Here, we sought additional evidence by using chatbots based on large language models as models of biology students. For human physiology and cell biology, we developed sets of realistic and hypothetical problems matched to the same lesson learning objectives (LLOs). Problems were considered hypothetical if (i) known biological entities (molecules and organs) were given atypical or counterfactual properties (redefinition) or (ii) fictitious biological entities were introduced (invention). Several chatbots scored significantly worse on hypothetical problems than on realistic problems, with scores declining by an average of 13%. Among hypothetical questions, redefinition questions appeared especially difficult, with many chatbots scoring as if guessing randomly. These results suggest that, for a given LLO, hypothetical problems may have different cognitive demands than realistic problems and may more accurately reveal students’ ability to apply biology core concepts to diverse contexts. The Test Question Templates (TQT) framework, which explicitly connects LLOs with examples of assessment questions, can help educators generate problems that are challenging (due to their novelty), yet fair (due to their alignment with pre-specified LLOs). Finally, ChatGPT’s rapid improvement toward expert-level answers suggests that future educators cannot reasonably expect to ignore or outwit chatbots but must do what we can to make assessments fair and equitable.
Metrics
Details
- Title
- Chatbot responses suggest that hypothetical biology questions are harder than realistic ones
- Creators
- Gregory J. Crowther - Everett Community CollegeUsha Sankar - Fordham UniversityLeena S. Knight - Whitman CollegeDeborah L. Myers - Everett Community CollegeKevin T. Patton - St. Charles Community CollegeLekelia D. Jenkins - Institute for the FutureThomas A. Knight - Whitman College
- Contributors
- Jack Wang (Editor)
- Publication Details
- Journal of microbiology & biology education, v 24(3)
- Publisher
- American Society for Microbiology
- Number of pages
- 14
- Resource Type
- Journal article
- Language
- English
- Academic Unit
- Biology
- Web of Science ID
- WOS:001095972600001
- Scopus ID
- 2-s2.0-85181822748
- Other Identifier
- 991021879779104721
UN Sustainable Development Goals (SDGs)
This publication has contributed to the advancement of the following goals:
InCites Highlights
Data related to this publication, from InCites Benchmarking & Analytics tool:
- Collaboration types
- Domestic collaboration
- Web of Science research areas
- Education, Scientific Disciplines