An Open Competition Involving Thousands of Competitors Failed to Construct Useful Abstract Classifiers for New Diagnostic Test Accuracy Systematic Reviews

Autor/inn/en	Kataoka, Yuki; Taito, Shunsuke; Yamamoto, Norio; So, Ryuhei; Tsutsumi, Yusuke; Anan, Keisuke; Banno, Masahiro; Tsujimoto, Yasushi; Wada, Yoshitaka; Sagami, Shintaro; Tsujimoto, Hiraku; Nihashi, Takashi; Takeuchi, Motoki; Terasawa, Teruhiko; Iguchi, Masahiro; Kumasawa, Junji; Ichikawa, Takumi; Furukawa, Ryuki; Yamabe, Jun; Furukawa, Toshi A.
Titel	An Open Competition Involving Thousands of Competitors Failed to Construct Useful Abstract Classifiers for New Diagnostic Test Accuracy Systematic Reviews
Quelle	In: Research Synthesis Methods, 14 (2023) 5, S.707-717 (11 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Kataoka, Yuki) ORCID (Taito, Shunsuke) ORCID (Yamamoto, Norio) ORCID (So, Ryuhei) ORCID (Tsutsumi, Yusuke) ORCID (Anan, Keisuke) ORCID (Banno, Masahiro) ORCID (Tsujimoto, Yasushi) ORCID (Wada, Yoshitaka) ORCID (Sagami, Shintaro) ORCID (Tsujimoto, Hiraku) ORCID (Kumasawa, Junji) ORCID (Furukawa, Toshi A.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1759-2879
DOI	10.1002/jrsm.1649
Schlagwörter	Competition; Classification; Diagnostic Tests; Accuracy; Artificial Intelligence; Documentation; Scores + Suchen Sie Ihr Suchwort? Wettkampf; Classification system; Klassifikation; Klassifikationssystem; Diagnostic test; Diagnostischer Test; Künstliche Intelligenz; Dokumentation
Abstract	There are currently no abstract classifiers, which can be used for new diagnostic test accuracy (DTA) systematic reviews to select primary DTA study abstracts from database searches. Our goal was to develop machine-learning-based abstract classifiers for new DTA systematic reviews through an open competition. We prepared a dataset of abstracts obtained through database searches from 11 reviews in different clinical areas. As the reference standard, we used the abstract lists that required manual full-text review. We randomly splitted the datasets into a train set, a public test set, and a private test set. Competition participants used the training set to develop classifiers and validated their classifiers using the public test set. The classifiers were refined based on the performance of the public test set. They could submit as many times as they wanted during the competition. Finally, we used the private test set to rank the submitted classifiers. To reduce false exclusions, we used the Fbeta measure with a beta set to seven for evaluating classifiers. After the competition, we conducted the external validation using a dataset from a cardiology DTA review. We received 13,774 submissions from 1429 teams or persons over 4 months. The top-honored classifier achieved a Fbeta score of 0.4036 and a recall of 0.2352 in the external validation. In conclusion, we were unable to develop an abstract classifier with sufficient recall for immediate application to new DTA systematic reviews. Further studies are needed to update and validate classifiers with datasets from other clinical areas. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01