Analysing academic paper ranking algorithms using test data and benchmarks: an investigation.

Autor/inn/en	Zhang, Yu; Wang, Min; Saberi, Morteza; Chang, Elizabeth
Titel	Analysing academic paper ranking algorithms using test data and benchmarks: an investigation.
Quelle	In: Scientometrics, (2022) 7, S.4045-4074 PDF als Volltext Verfügbarkeit
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0138-9130
DOI	10.1007/s11192-022-04429-z
Schlagwörter	Academic paper ranking; Ranking algorithms; Bibliometric analysis; Citation analysis
Abstract	Abstract Research on academic paper ranking has received great attention in recent years, and many algorithms have been proposed to automatically assess a large number of papers for this purpose. How to evaluate or analyse the performance of these ranking algorithms becomes an open research question. Theoretically, evaluation of an algorithm requires to compare its ranking result against a ground truth paper list. However, such ground truth does not exist in the field of scholarly ranking due to the fact that there does not and will not exist an absolutely unbiased, objective, and unified standard to formulate the impact of papers. Therefore, in practice researchers evaluate or analyse their proposed ranking algorithms by different methods, such as using domain expert decisions (test data) and comparing against predefined ranking benchmarks. The question is whether using different methods leads to different analysis results, and if so, how should we analyse the performance of the ranking algorithms? To answer these questions, this study compares among test data and different citation-based benchmarks by examining their relationships and assessing the effect of the method choices on their analysis results. The results of our experiments show that there does exist difference in analysis results when employing test data and different benchmarks, and relying exclusively on one benchmark or test data may bring inadequate analysis results. In addition, a guideline on how to conduct a comprehensive analysis using multiple benchmarks from different perspectives is summarised, which can help provide a systematic understanding and profile of the analysed algorithms.
Erfasst von	OLC
Update	2023/2/05