Modeling and Analyzing Scorer Preferences in Short-Answer Math Questions [Konferenzbericht] Paper presented at the International Conference on Educational Data Mining (EDM) (16th, Bengaluru, India, Jul 11-14, 2023).

Autor/inn/en	Zhang, Mengxue; Heffernan, Neil; Lan, Andrew
Titel	Modeling and Analyzing Scorer Preferences in Short-Answer Math Questions [Konferenzbericht] Paper presented at the International Conference on Educational Data Mining (EDM) (16th, Bengaluru, India, Jul 11-14, 2023).
Quelle	(2023), (10 Seiten) PDF als Volltext kostenfreie Datei Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Monographie
Schlagwörter	Scoring; Computer Assisted Testing; Mathematics Instruction; Mathematics Tests; Computational Linguistics; Computer Software; Evaluators; Models; Test Items; Accuracy; Case Studies; Preferences; Item Analysis; Comparative Analysis + Suchen Sie Ihr Suchwort? Bewertung; Mathematics lessons; Mathematikunterricht; Linguistics; Computerlinguistik; Analogiemodell; Test content; Testaufgabe; Case study; Fallstudie; Case Study; Itemanalyse
Abstract	Automated scoring of student responses to open-ended questions, including short-answer questions, has great potential to scale to a large number of responses. Recent approaches for automated scoring rely on supervised learning, i.e., training classifiers or fine-tuning language models on a small number of responses with human-provided score labels. However, since scoring is a subjective process, these human scores are noisy and can be highly variable, depending on the scorer. In this paper, we investigate a collection of models that account for the individual preferences and tendencies of each human scorer in the automated scoring task. We apply these models to a short-answer math response dataset where each response is scored (often differently) by multiple different human scorers. We conduct quantitative experiments to show that our scorer models lead to improved automated scoring accuracy. We also conduct quantitative experiments and case studies to analyze the individual preferences and tendencies of scorers. We found that scorers can be grouped into several obvious clusters, with each cluster having distinct features, and analyzed them in detail. [For the complete proceedings, see ED630829.] (As Provided).
Anmerkungen	International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01