Scoring Graphical Responses in TIMSS 2019 Using Artificial Neural Networks

Autor/inn/en	von Davier, Matthias; Tyack, Lillian; Khorramdel, Lale
Titel	Scoring Graphical Responses in TIMSS 2019 Using Artificial Neural Networks
Quelle	In: Educational and Psychological Measurement, 83 (2023) 3, S.556-585 (30 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (von Davier, Matthias)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0013-1644
DOI	10.1177/00131644221098021
Schlagwörter	Scoring; Networks; Artificial Intelligence; Elementary Secondary Education; Foreign Countries; Achievement Tests; International Assessment; Mathematics Tests; Mathematics Achievement; Science Achievement; Science Tests; Classification; Test Items; Item Analysis; Responses; Accuracy; Models; Evaluators; Comparative Analysis; Item Response Theory; Test Validity; Graphs; Computer Software; Trends in International Mathematics and Science Study + Suchen Sie Ihr Suchwort? Bewertung; Künstliche Intelligenz; Ausland; Achievement test; Achievement; Testing; Test; Tests; Leistungsbeurteilung; Leistungsüberprüfung; Leistung; Testdurchführung; Testen; Mathmatics sikills; Mathmatics achievement; Mathematical ability; Mathematische Kompetenz; Classification system; Klassifikation; Klassifikationssystem; Test content; Testaufgabe; Itemanalyse; Analogiemodell; Item-Response-Theorie; Testvalidität; Grafische Darstellung
Abstract	Automated scoring of free drawings or images as responses has yet to be used in large-scale assessments of student achievement. In this study, we propose artificial neural networks to classify these types of graphical responses from a TIMSS 2019 item. We are comparing classification accuracy of convolutional and feed-forward approaches. Our results show that convolutional neural networks (CNNs) outperform feed-forward neural networks in both loss and accuracy. The CNN models classified up to 97.53% of the image responses into the appropriate scoring category, which is comparable to, if not more accurate, than typical human raters. These findings were further strengthened by the observation that the most accurate CNN models correctly classified some image responses that had been incorrectly scored by the human raters. As an additional innovation, we outline a method to select human-rated responses for the training sample based on an application of the expected response function derived from item response theory. This paper argues that CNN-based automated scoring of image responses is a highly accurate procedure that could potentially replace the workload and cost of second human raters for international large-scale assessments (ILSAs), while improving the validity and comparability of scoring complex constructed-response items. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01