AI ‐Augmented Search for Systematic Reviews: A Comparative Analysis
Abstract
ABSTRACT Researchers are increasingly seeking to automate systematic review workflows to reduce time and labor. However, AI‐generated search strategies often contain errors that can significantly undermine the reliability of the evidence. To evaluate the relevancy and reproducibility of automated search strategies, we conducted a comparative analysis between a human‐in‐the‐loop system with a neurosymbolic AI framework (i.e., NeuroLit Navigator) and three AI systems that primarily rely on generative language models: Scite, Consensus, and Perplexity. Results showed that through an iterative human‐in‐the‐loop approach, NeuroLit Navigator produced more precise and focused search strategies aligned with domain‐specific terminologies compared to commercial LLM‐based systems, which presented issues such as lack of reproducibility and interpretability. This study highlights the potential of human‐AI collaboration in systematic review workflows, suggesting that AI should augment, rather than replace, librarian expertise. Our findings contribute to the growing field of human‐centered AI by providing a model for designing AI systems in information‐intensive domains.
How strong is the evidence?
Evidence Hub reads this as a systematic review. Its evidence level is 1 of 5 (Highest · pooled evidence). Pooled evidence from several studies sits at the top of the evidence pyramid.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.