The Noisy Label Filter procedure: a case study to address replication issues in systematic reviews

Filtering noisy-labeled records into relevant and irrelevant, the Noisy Label Filter (NLF) procedure could provide a solution to replication issues in systematic review’s datasets.

Originating as a single-student project in 2022, this article has evolved into a multidisciplinary manuscript by 2023. Currently, it is being reviewed by the journal ‘Systematic Reviews’ and has already been released as a preprint.

Within this study:

  1. We reconstructed a dataset containing both included and excluded records from a previously published systematic review on the treatment of borderline personality disorder,
  2. We developed and performed a procedure to assign accurate labels to records with unknown (noisy) labels: known as “the Noisy Label Filter (NLF) procedure”.
  3. Using ASReview Makita, we conducted a simulation study with the final reconstructed dataset. We found out how much time could have been saved using active learning in the original study.

The main issue discussed in this paper is that correctly following the Prisma guidelines, as did our use case study, does not imply full replicability. It takes a lot of time and effort to reproduce the initial dataset, knowing that the reconstruction will probably be imperfect. Following the initial inclusion criteria, the NLF procedure can help to assign correct labels to the reconstructed dataset. However, one can never be sure if this dataset exactly matches the initial unstored dataset. This can be due to mismatched results in the closed-source search tools, retracted papers, and several other reasons. A possible solution is to store the completely labeled dataset of included and excluded studies initially.


The article has been published as a preprint.

Neeleman, R. C., Leenaars, C., Oud, M., Weijdema, F., & van de Schoot, R. (2023, August 3). Addressing the Challenges of Reconstructing Systematic Reviews Datasets.


Scripts, data and output concerning this study

To reproduce this study, scripts, data, and output are stored on OSF (open science framework). 

Neeleman, R. C., Oud, M., Weijdema, F., Leenaars, C., & van de Schoot, R. (2023, June 22). Scripts, data and output to reproduce “Addressing the Challenges of Reconstructing Systematic Reviews Datasets: A Case Study and a Noisy Label Filter Procedure.”

Scripts, data and output

Initial study by Oud et al. (2018)

The original study was performed by Oud et al. (2018), who systematically reviewed the treatment of Borderline Personality Disorder. We tried to reconstruct their dataset of included and excluded records.

Oud, M., Arntz, A., Hermens, M. L., Verhoef, R., & Kendall, T. (2018). Specialized psychotherapies for adults with borderline personality disorder: A systematic review and meta-analysis. Australian & New Zealand Journal of Psychiatry, 52(10), 949–961.

Initial study (Oud et al., 2018)