Letter to the Editor: Refining retrieval and chunking strategies for enhanced clinical reliability of large language models in liver disease
Abstract
To the editor, I have read with interest the study by Ge et al.1 This work marks a significant milestone in leveraging large language models (LLMs) for clinical applications, notably for its pioneering effort to create a Personal Health Information-compliant liver disease–specific model. Despite the innovation, certain areas warrant further clarification and improvement to fully understand the impact and reliability of the framework proposed by the authors. First and foremost, it is crucial to have transparent information regarding the graders' level of expertise and to address the risks of performance evaluation associated with the use of grading scales, rather than binary grading, as discussed elsewhere.2,3 It has been notably intriguing to observe that the Retrieval Augmented Generation (RAG) yielded responses significantly more accurate than those generated by the foundational GPT-4 model in merely 20% of the topics addressed (ie, HCC and DILI).1 Compared to other RAG applications in digestive disease,4 this performance discrepancy prompts questions about the data quality used for RAG, particularly the conversion of medical guidelines into text for the model. The potential inaccuracies from standard text converters and GPT-Vision in parsing complex formats (eg, graphical tables or flowcharts) containing critical information could significantly impact response accuracy. In addition, according to the reported context window, I believe the authors adopted text chunking,…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.