Artificial intelligence supports literature screening in medical guideline development: towards up-to-date medical guidelines
Harmsen, Wouter; de Groot, Janke; Harkema, Albert; van Dusseldorp, Ingeborg; De Bruin, Jonathan; Van den Brand, Sofie; Van de Schoot, Rens
Objectives: In a time of exponential growth of new evidence supporting clinical decision making, combined with a labor-intensive process of selecting this evidence, there is a need for methods to speed up current processes in order to keep medical guidelines up-to-date. The purpose of this study was to evaluate the performance and feasibility of active learning to support the selection of relevant publications within the context of medical guideline development.
Design: We used a mixed methods design. The manual process of literature selection by two independent clinicians was evaluated in 14 searches by calculating Cohen’s Kappa (ĸ) for interrater reliability. This was followed by a series of simulations investigating the performance of random reading versus using screening prioritization based on active learning.
Main outcome measures: Work Saved over Sampling at 95% recall (WSS@95), percentage Relevant Records Found at reading only 10% of the total number of records (RRF@10) and average time to discovery (ATD). Finally, results were discussed in a reflective dialogue with guideline developers.
Results: Mean ĸ for manual title-abstract selection by clinicians was 0.50 and varied between -0.01 to 0.87 based on 5021 abstracts. WSS@95 ranged from 50.15% (SD=17.7) based on selection by clinicians, to 69.24% (SD=11.5) based on the selection by research methodologist up to 75.76% (SD=12.2) based on the final full-text inclusion. A similar pattern was seen for RRF@10 ranging from 48.31% (SD= 23.3) to 62.8% (SD=21.20) and to 65.58% (SD=23.25). ATD ranged from 20 to 67 abstracts.
Conclusion: Tools, implementing active learning, such as ASReview, can speed up the process of literature screening within guideline development.
Harmsen, Wouter, de Groot, Janke, Harkema, Albert, van Dusseldorp, Ingeborg, De Bruin, Jonathan, Van den Brand, Sofie, & Van de Schoot, Rens. (2021). Artificial intelligence supports literature screening in medical guideline development: towards up-to-date medical guidelines. (Version V1.0). Zenodo. DOI: 10.5281/zenodo.5031907