Towards Trustworthy Autograding of Short, Multi-Lingual, Multi-Type Answers

Autor/inn/en	Schneider, Johannes; Richner, Robin; Riser, Micha
Titel	Towards Trustworthy Autograding of Short, Multi-Lingual, Multi-Type Answers
Quelle	In: International Journal of Artificial Intelligence in Education, 33 (2023) 1, S.88-118 (31 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1560-4292
DOI	10.1007/s40593-022-00289-z
Schlagwörter	Grading; Natural Language Processing; Computer Assisted Testing; Ethics; Syntax; Computational Linguistics; Accuracy; Trust (Psychology); Error Patterns; Error Correction; Evaluation Methods + Suchen Sie Ihr Suchwort? Notengebung; Schulnote; Natürliche Sprache; Ethik; Linguistics; Computerlinguistik; Fehlertyp; Korrektur
Abstract	Autograding short textual answers has become much more feasible due to the rise of NLP and the increased availability of question-answer pairs brought about by a shift to online education. Autograding performance is still inferior to human grading. The statistical and black-box nature of state-of-the-art machine learning models makes them untrustworthy, raising ethical concerns and limiting their practical utility. Furthermore, the evaluation of autograding is typically confined to small, monolingual datasets for a specific question type. This study uses a large dataset consisting of about 10 million question-answer pairs from multiple languages covering diverse fields such as math and language, and strong variation in question and answer syntax. We demonstrate the effectiveness of fine-tuning transformer models for autograding for such complex datasets. Our best hyperparameter-tuned model yields an accuracy of about 86.5%, comparable to the state-of-the-art models that are less general and more tuned to a specific type of question, subject, and language. More importantly, we address trust and ethical concerns. By involving humans in the autograding process, we show how to improve the accuracy of automatically graded answers, achieving accuracy equivalent to that of teaching assistants. We also show how teachers can effectively control the type of errors made by the system and how they can validate efficiently that the autograder's performance on individual exams is close to the expected performance. (As Provided).
Anmerkungen	Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01