The Noisy Label Filter procedure: a case study to address replication issues in systematic reviews

Filtering noisy-labeled records into relevant and irrelevant, the Noisy Label Filter (NLF) procedure could provide a solution to replication issues in systematic review’s datasets.

Originating as a single-student project in 2022, this article has evolved into a multidisciplinary manuscript by 2023. Currently, it is being reviewed by the journal ‘Systematic Reviews’ and has already been released as a preprint.

Within this study:

1. We reconstructed a dataset containing both included and excluded records from a previously published systematic review on the treatment of borderline personality disorder,
2. We developed and performed a procedure to assign accurate labels to records with unknown (noisy) labels: known as “the Noisy Label Filter (NLF) procedure”.
3. Using ASReview Makita, we conducted a simulation study with the final reconstructed dataset. We found out how much time could have been saved using active learning in the original study.

The main issue discussed in this paper is that correctly following the Prisma guidelines, as did our use case study, does not imply full replicability. It takes a lot of time and effort to reproduce the initial dataset, knowing that the reconstruction will probably be imperfect. Following the initial inclusion criteria, the NLF procedure can help to assign correct labels to the reconstructed dataset. However, one can never be sure if this dataset exactly matches the initial unstored dataset. This can be due to mismatched results in the closed-source search tools, retracted papers, and several other reasons. A possible solution is to store the completely labeled dataset of included and excluded studies initially.