A minimal bias, truly Human-in-the-Loop system
Every so often a discussion item pops up on the ASReview GitHub page with the suggestion that the interface would be even better if it also showed the likelihood that an abstract is relevant, or the journal a record was published in. Sixu Cai, PhD candidate at the DISC-AI Lab of Utrecht University, investigated what the effect of showing this information would be on the objectivity of the Oracle, our human in the loop. This blog post explains his findings and why ASReview LAB kept the lean-and-mean interface you know and love.
Do you want to read the full paper? You can find the preprint on ResearchGate!
Research Design
Sixu’s research on human factors in AI-aided abstract screening shows that people’s screening decisions are strongly influenced by signals that are not part of the abstract content itself¹. He investigated the effects of showing two different types of information: 1) whether an AI-agent recommended an abstract as relevant or not, and 2) in which journal the paper had been published.
In the study, the AI-recommendation was operationalised as a soft suggestion, phrased along the lines of “the AI recommends including this abstract” or “the AI recommends not including this abstract.” Despite this cautious wording, users were already biased by the presence of the AI signal. They were more likely to follow the AI’s suggestion instead of relying fully on their own reading, even when the suggestion was incorrect.
This increases the risk of both types of errors: people include papers they should not include, and—more seriously—exclude papers that should have been included. Once a relevant paper is excluded at the abstract screening stage, it is usually lost for good.
Similarly, we found that displaying journal information can also bias decisions. Inherently, journal names say nothing about whether a paper is relevant to a specific review or not. However, when journal names were shown, users proved to be more likely to include papers simply because they come from well-known journals, even when the abstract was in fact irrelevant to the study. This means that relevance decisions are no longer based purely on content, but on authoritative cues that are irrelevant to the task.
¹ Cai, S., van Roekel, H., Tummers, L., & van de Schoot, R. (2025, December 17). Authority Bias in Human-AI Decision Making: The Effects of AI Appraisals and Journal Cues in Abstract Screening. https://doi.org/10.31234/osf.io/57my4_v1
Conclusion
Taken together, the evidence indicates that the more extra signals a system shows, the more likely users are to drift away from content-based screening.
For this reason, ASReview’s design choice to:
- keep humans in control (Human-in-the-Loop),
- avoid AI-recommended rating systems, and
- limit peripheral information such as journal names,
is not just a design preference, but a way to reduce systematic bias and protect the quality of the screening process.
