Making the Cut: How Panel Reviewers Use Evaluation Devices to Select Applications at the European Research Council

Autor/inn/en	Brunet, Lucas; Müller, Ruth
Titel	Making the Cut: How Panel Reviewers Use Evaluation Devices to Select Applications at the European Research Council
Quelle	In: Research Evaluation, 31 (2022) 4, S.486-497 (12 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Brunet, Lucas)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0958-2029
DOI	10.1093/reseval/rvac040
Schlagwörter	Peer Evaluation; Evaluation Methods; International Organizations; Evaluation Utilization; Evaluators; Expertise; Research Proposals; Governance; Foreign Countries; Europe + Suchen Sie Ihr Suchwort? International organisation; International organisations; International organization; Internationale Organisation; Expert appraisal; Forschungsgegenstand; Education; Educational policy; Financing; Steuerung; Bildung; Erziehung; Bildungspolitik; Finanzierung; Ausland; Europa
Abstract	The European Research Council (ERC) receives many high-quality applications, but funds only a few. We analyze how members of ERC review panels assess applications in the first, highly competitive step of evaluations for ERC Starting and Consolidator Grants. Drawing on interviews with ERC panel members in different fields, we show that they adopt a set of "evaluation devices" that offer pragmatic and standardized ways of evaluating in a time-constrained and highly competitive setting. Through the use of evaluation devices, panel reviewers enact and generate a distinct "reviewing expertise" that encompasses subject-specific knowledge and knowledge about how to accomplish evaluation within a situated setting. We find that ERC panel reviewers employ four evaluation devices during the first step of ERC reviews: first, reviewers base judgments on applicants' prior achievements ("delegation devices"); second, they adjust their evaluations of individual applications to the quality of a given set of applications ("calibration devices"); third, they combine multiple elements to assess the feasibility of proposals ("articulation devices"); and finally, they consider the impact of the proposed research on science and society ("contribution devices"). We show that the current use of these devices generates what we have termed "evaluative pragmatism": a mode of reviewing that is shaped by and accommodated to the need to review many high-quality proposals in a short time period with possibly limited expert knowledge. In conclusion, we discuss how the prevalence of evaluative pragmatism in the first step of ERC panel reviews shapes candidate selection, particularly regarding human and epistemic diversity in European research. (As Provided).
Anmerkungen	Oxford University Press. Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44-1865-353907; Fax: +44-1865-353485; e-mail: jnls.cust.serv@oxfordjournals.org; Web site: http://applij.oxfordjournals.org/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01