Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer-Review Models

Autor/inn/en	Darvishi, Ali; Khosravi, Hassan; Rahimi, Afshin; Sadiq, Shazia; Gasevic, Dragan
Titel	Assessing the Quality of Student-Generated Content at Scale: A Comparative Analysis of Peer-Review Models
Quelle	In: IEEE Transactions on Learning Technologies, 16 (2023) 1, S.106-120 (15 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Darvishi, Ali) ORCID (Khosravi, Hassan) ORCID (Sadiq, Shazia) ORCID (Gasevic, Dragan)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
DOI	10.1109/TLT.2022.3229022
Schlagwörter	Student Developed Materials; Educational Assessment; Peer Evaluation; Evaluation Methods; Educational Quality; Reliability; Evaluators; Learning Analytics + Suchen Sie Ihr Suchwort? Education; assessment; Bewertungssystem; Quality of education; Bildungsqualität; Reliabilität
Abstract	Engaging students in creating learning resources has demonstrated pedagogical benefits. However, to effectively utilize a repository of student-generated content (SGC), a selection process is needed to separate high- from low-quality resources as some of the resources created by students can be ineffective, inappropriate, or incorrect. A common and scalable approach is to use a peer-review process where students are asked to assess the quality of resources authored by their peers. Given that judgments of students, as experts-in-training, cannot wholly be relied upon, a redundancy-based method is widely employed where the same assessment task is given to multiple students. However, this approach introduces a new challenge, referred to as the consensus problem: How can we assign a final quality to a resource given ratings by multiple students? To address this challenge, we investigate the predictive performance of 18 inference models across five well-established categories of consensus approaches for inferring the quality of SGC at scale. The analysis is based on the engagement of 2141 undergraduate students across five courses in creating 12 803 resources and 77 297 peer reviews. Results indicate that the quality of reviews is quite diverse, and students tend to overrate. Consequently, simple statistics such as mean and median fail to identify poor-quality resources. Findings further suggest that incorporating advanced probabilistic and text analysis methods to infer the reviewers' reliability and reviews' quality improves performance; however, there is still an evident need for instructor oversight and training of students to write compelling and reliable reviews. (As Provided).
Anmerkungen	Institute of Electrical and Electronics Engineers, Inc. 445 Hoes Lane, Piscataway, NJ 08854. Tel: 732-981-0060; Web site: http://ieeexplore.ieee.org/xpl/RecentIssue.jsp?punumber=4620076
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01