Logo image
Back
Scaling Retrieval Practice with LLM: Improving Multiple Choice Question (MCQ) Quality through Knowledge Graphs
Conference paper   Open access

Scaling Retrieval Practice with LLM: Improving Multiple Choice Question (MCQ) Quality through Knowledge Graphs

Yuan An and Ruhma Hashmi
Proceedings of the 57th ACM Technical Symposium on Computer Science Education V.2, pp 1221-1222
18 Feb 2026
DOI: https://doi.org/10.1145/3770761.3777185
url
https://doi.org/10.1145/3770761.3777185View
Published, Version of Record (VoR) Open CC BY-NC-ND V4.0

Abstract

Information systems -- Expert systems
Teaching introductory computer science has become increasingly difficult with the rise of AI code-completion tools. Frequent retrieval practice, especially through multiple-choice questions (MCQs), offers a promising way to maintain active learning, yet producing high quality MCQs at scale remains a challenge for instructors. Large language models (LLMs) can automate MCQ generation, enabling scalable in-lecture retrieval practice. This poster presents two preliminary studies examining this potential. First, in higher-education programming courses, students who received LLM-generated MCQs scored significantly higher on follow-up quizzes than during periods without retrieval practice. However, raw LLM generated MCQs often suffered from hallucinations, weak distractors, trivial content, and formatting issues. Second, we evaluated a knowledge graph (KG)–guided generation pipeline. By structuring key concepts and relations before prompting, the KG-based approach produced more relevant, integrative, and appropriately challenging MCQs. In a dataset of 400+ items, KG-generated MCQs outperformed text-only generation across 15 quality criteria. These preliminary studies open promising directions for future research, including adaptive generation of personalized MCQs tailored to student mastery levels.

Metrics

1 Record Views

Details

Logo image