Evaluating LLM-Based Retrieval-Augmented Generation for Soil Science Question Answering
Abstract
Retrieval-augmented generation (RAG) systems for scientific literature require evidence-based choices of document segmentation, representation, retrieval, and generation components, particularly when the source collection varies in topical specificity and document structure. This study addresses the lack of an end-to-end, component-level comparison of these choices for soil science question answering. A three-stage evaluation was conducted across general, domain-specific, and geospatial soil science corpora. The corpus combines foundational soil science books, peer-reviewed research articles, European soil monitoring material, and geospatial mapping publications, thereby covering both broad disciplinary concepts and specialized scientific evidence. The study compares four chunking strategies, three embedding models, five retrieval methods, and five large language models. In Experiment 1, semantic chunking with text-embedding-3-large achieved the highest aggregate retrieval scores (recall@1 = 0.824; MRR = 0.819), whereas text-embedding-3-small delivered practically comparable performance at lower cost. In Experiment 2, hybrid reciprocal rank fusion achieved recall@5 values of 0.957, 0.960, and 0.647 for the general, domain-specific, and geospatial corpora, respectively; the cross-encoder reranker showed weaker rank quality on scientific content. In Experiment 3, model responses attained BERTScore values of 0.909–0.927 and faithfulness of at least 0.993; these automated…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.