Judges' Agreement and Disagreement Patterns When Encoding Verbal Protocols.

Autor/inn/en	Schael, Jocelyne; Dionne, Jean-Paul
Titel	Judges' Agreement and Disagreement Patterns When Encoding Verbal Protocols.
Quelle	(1991), (25 Seiten)Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; Monographie
Schlagwörter	Adults; College Graduates; Comparative Analysis; Encoding (Psychology); Evaluators; Interrater Reliability; Nonverbal Communication; Protocol Analysis; Sex Differences + Suchen Sie Ihr Suchwort? Hochschulabsolvent; Hochschulabsolventin; Encoding; Codierung; Interrater-Reliabilität; Non-verbal communication; Nonverbale Kommunikation; Sex difference; Geschlechtsunterschied
Abstract	The basis of agreement or disagreement among judges/evaluators when applying a coding scheme to concurrent verbal protocols was studied. The sample included 20 university graduates, from varied backgrounds; 10 subjects had and 10 subjects did not have experience in protocol analysis. The total sample was divided into four balanced groups according to gender and experience in protocol analysis. Subjects were asked to think aloud while encoding two protocols--extracted from an ill-designed task--with a grid of categories provided to them. Subjects' verbalizations and non-verbal behaviors and manipulations were recorded. The level of agreement on the final decisions was analyzed under several criteria. The overall agreement among subjects on the selected codes was 61.4%, but the agreement on the reasons for choosing these codes was 85.1%. This marked difference indicates that the codes selected by the evaluators were not always supported by the reasons stated or that the grid categories are ambiguous. The results show that experience with the task presented (i.e., planning a three-course meal) seemed to be more important for agreement than experience with the technique of protocol analysis. Criteria for improving the quality of a grid and the reliability of coders are suggested. Three data tables and two bar graphs are included. The two study protocols and the coding grid are presented. (Author/RLC)
Erfasst von	ERIC (Education Resources Information Center), Washington, DC