Towards Automatic Annotation of Collaborative Problem-Solving Skills in Technology-Enhanced Environments

Autor/inn/en	Flor, Michael; Andrews-Todd, Jessica
Titel	Towards Automatic Annotation of Collaborative Problem-Solving Skills in Technology-Enhanced Environments
Quelle	In: Journal of Computer Assisted Learning, 38 (2022) 5, S.1434-1447 (14 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Flor, Michael)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0266-4909
DOI	10.1111/jcal.12689
Schlagwörter	Automation; Documentation; Cooperative Learning; Teamwork; Problem Solving; Technology Uses in Education; Educational Assessment; Electronic Learning; Artificial Intelligence; Man Machine Systems; Classification; Accuracy + Suchen Sie Ihr Suchwort? Dokumentation; Kooperatives Lernen; Problemlösen; Technology enhanced learning; Technology aided learning; Technologieunterstütztes Lernen; Education; assessment; Bewertungssystem; Künstliche Intelligenz; Mensch-Maschine-System; Classification system; Klassifikation; Klassifikationssystem
Abstract	Background: Collaborative problem solving (CPS) is important for success in the 21st century, especially for teamwork and communication in technology-enhanced environments. Measurement of CPS skills has emerged as an essential aspect in educational assessment. Modern research in CPS relies on theory-driven measurements that are usually carried out as manual annotations over recorded logs of collaborative activities. However, manual annotation has limited scalability and is not conductive towards CPS assessments at scale. Objective: We explore possibilities for automated annotation of actions in collaborative-teams, especially chat messages. We evaluate two approaches that employ machine learning for automated classification of CPS events. Method: Data were collected from engineering, physics and electronics students' participation in a simulation-based task on electronics concepts, in which participants communicated via text-chat messages. All task activities were logged and time stamped. Data have been manually classified for the CPS skills, using an ontology that includes both social and cognitive dimensions. In this article, we describe computational linguistic methods for automatically classifying the CPS skills from logged data, with a view towards automating CPS assessments. Results: We applied two machine learning methods to our data. A Naïve Bayes classifier has been previously used in CPS research, but it is only moderately successful on our data. We also present a k-nearest-neighbours (kNN) classifier that uses distributional semantic models for measuring text similarity. This classifier shows strong agreement between automated and human annotations. The study also demonstrates that automatic spelling correction and slang normalization of chat texts are useful for accurate automated annotation. Implications: Our results suggest that a kNN classifier can be very effective for accurate annotation of CPS events. It achieves reasonably strong results even when trained on only half of the available data. This shows a promise towards reduction of manual data annotation for CPS measurement. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01