Artificial intelligence supports literature screening in medical guideline development: towards up-to-date medical guidelines

Harmsen, Wouter; de Groot, Janke; Harkema, Albert; van Dusseldorp, Ingeborg; De Bruin, Jonathan; Van den Brand, Sofie; Van de Schoot, Rens

Abstract

Objectives: In a time of exponential growth of new evidence supporting clinical decision making, combined with a labor-intensive process of selecting this evidence, there is a need for methods to speed up current processes in order to keep medical guidelines up-to-date. The purpose of this study was to evaluate the performance and feasibility of active learning to support the selection of relevant publications within the context of medical guideline development.

Design: We used a mixed methods design. The manual process of literature selection by two independent clinicians was evaluated in 14 searches by calculating Cohen’s Kappa (ĸ) for interrater reliability. This was followed by a series of simulations investigating the performance of random  reading versus using screening prioritization based on active learning.

Main outcome measures: Work Saved over Sampling at 95% recall (WSS@95), percentage Relevant Records Found at reading only 10% of the total number of records (RRF@10) and average time to discovery (ATD). Finally, results were discussed in a reflective dialogue with guideline developers.

Results: Mean ĸ for manual title-abstract selection by clinicians was 0.50 and varied between -0.01 to 0.87 based on 5021 abstracts. WSS@95 ranged from 50.15% (SD=17.7) based on selection by clinicians, to 69.24% (SD=11.5) based on the selection by research methodologist up to 75.76% (SD=12.2) based on the final full-text inclusion. A similar pattern was seen for RRF@10 ranging from 48.31% (SD= 23.3) to 62.8% (SD=21.20) and to 65.58% (SD=23.25). ATD ranged from 20 to 67 abstracts.

Conclusion: Tools, implementing active learning, such as ASReview, can speed up the process of literature screening within guideline development.

 

Harmsen, Wouter, de Groot, Janke, Harkema, Albert, van Dusseldorp, Ingeborg, De Bruin, Jonathan, Van den Brand, Sofie, & Van de Schoot, Rens. (2021). Artificial intelligence supports literature screening in medical guideline development: towards up-to-date medical guidelines. (Version V1.0). Zenodo. DOI: 10.5281/zenodo.5031907

Using ASReview on your tablet
Data
Scripts
Pre-print
Categories: Data, Scientific Papers

Check out similar projects

Optimizing ASReview simulations

Optimizing ASReview simulations with multiprocessing solutions for ‘light-data’ and ‘heavy-data’ users via a Kubernetes cluster.

The FORAS project

The FORAS project will replicate and extend an original review integrating advanced machine-learning techniques via the OpenAlex database.

The Mega Meta project: Substance use, anxiety and depression

The MegaMeta project is a large scale project to review factors that contribute to substance use, anxiety and depressive disorders. Read more information on the search and screening protocol, hyperparameter tuning and post-processing used in this post.

ASReview paper explosion

Paper introducing the ASReview project

We show that by using active learning, ASReview can lead to far more efficient reviewing than manual reviewing, while exhibiting adequate quality. Furthermore, the presented software is fully transparent and open source.

Systematic review data about depression

Explore the systematic review dataset that was used for the publication “Psychological theories of depressive relapse and recurrence” from Brouwer et al., 2019. From pre-processing to the final dataset, a look into the complete systematic review process behind this publication.