Assessing the Validity of Test Scores Using Response Process Data from an Eye-Tracking Study: A New Approach

Autor/inn/en	Yaneva, Victoria; Clauser, Brian E.; Morales, Amy; Paniagua, Miguel
Titel	Assessing the Validity of Test Scores Using Response Process Data from an Eye-Tracking Study: A New Approach
Quelle	In: Advances in Health Sciences Education, 27 (2022) 5, S.1401-1422 (22 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Yaneva, Victoria)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1382-4996
DOI	10.1007/s10459-022-10107-9
Schlagwörter	Eye Movements; Artificial Intelligence; Scores; Test Interpretation; Multiple Choice Tests; Prediction; Validity + Suchen Sie Ihr Suchwort? Augenbewegung; Künstliche Intelligenz; Test analysis; Testauswertung; Multiple choice examinations; Multiple-choice tests, Multiple-choice examinations; Multiple-Choice-Verfahren; Vorhersage; Gültigkeit
Abstract	Understanding the response process used by test takers when responding to multiple-choice questions (MCQs) is particularly important in evaluating the validity of score interpretations. Previous authors have recommended eye-tracking technology as a useful approach for collecting data on the processes test taker's use to respond to test questions. This study proposes a new method for evaluating alternative score interpretations by using eye-tracking data and machine learning. We collect eye-tracking data from 26 students responding to clinical MCQs. Analysis is performed by providing 119 eye-tracking features as input for a machine-learning model aiming to classify correct and incorrect responses. The predictive power of various combinations of features within the model is evaluated to understand how different feature interactions contribute to the predictions. The emerging eye-movement patterns indicate that incorrect responses are associated with working from the options to the stem. By contrast, correct responses are associated with working from the stem to the options, spending more time on reading the problem carefully, and a more decisive selection of a response option. The results suggest that the behaviours associated with correct responses are aligned with the real-world model used for score interpretation, while those associated with incorrect responses are not. To the best of our knowledge, this is the first study to perform data-driven, machine-learning experiments with eye-tracking data for the purpose of evaluating score interpretation validity. (As Provided).
Anmerkungen	Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01