Logo image
Back
Flip it: An exploratory (versus explanatory) sequential mixed methods design using Delphi and differential item functioning to evaluate item bias
Journal article   Open access   Peer reviewed

Flip it: An exploratory (versus explanatory) sequential mixed methods design using Delphi and differential item functioning to evaluate item bias

Kristin L.K. Koskey, Toni A. May, Yiyun “Kate” Fan, Dara Bright, Gregory Stone, Gabriel Matney and Jonathan D. Bostic
Methods in Psychology (Online), v 8, 100117
01 Nov 2023
DOI: https://doi.org/10.1016/j.metip.2023.100117
url
https://doi.org/10.1016/j.metip.2023.100117View
Published, Version of Record (VoR)CC BY-NC-ND V4.0,  Open

Abstract

Concordance Delphi DIF Exploratory sequential Item bias Joint display
The Delphi method has been adapted to inform item refinements in educational and psychological assessment development. An explanatory sequential mixed methods design using Delphi is a common approach to gain experts' insight into why items might have exhibited differential item functioning (DIF) for a sub-group, indicating potential item bias. Use of Delphi before quantitative field testing to screen for potential sources leading to item bias is lacking in the literature. An exploratory sequential design is illustrated as an additional approach using a Delphi technique in Phase I and Rasch DIF analyses in Phase II. We introduce the 2 × 2 Concordance Integration Typology as a systematic way to examine agreement and disagreement across the qualitative and quantitative findings using a concordance joint display table. A worked example from the development of the Problem-Solving Measures Grades 6–8 Computer Adaptive Tests supported using an exploratory sequential design to inform item refinement. The 2 × 2 Concordance Integration Typology (a) crystallized instances where additional refinements were potentially needed and (b) provided for evaluating the distribution of bias across the set of items as a whole. Implications are discussed for advancing data integration techniques and using mixed methods to improve instrument development.

Metrics

23 Record Views
6 citations in Scopus

Details

Logo image