Evaluating Local LLMs and Retrieval-Augmented Generation for Requirement Checking in Research Funding Applications
Abstract
Large Language Models (LLMs) have created new opportunities for automating document-intensive tasks, but cloud-based systems are often unsuitable in organizations handling sensitive data. This has increased interest in lower-parameter LLMs suitable for local execution, which offer stronger data control but operate under stricter computational constraints. This thesis investigates whether locally executed LLMs can support requirement checking in research funding applications at the Swedish Foundation for Strategic Research (SSF). To enable document analysis within limited context windows, a Retrieval-Augmented Generation (RAG) pipeline was implemented and evaluated. In particular, a naive retrieval approach was compared with an improved Parent-Document Retrieval (PDR) strategy. Several locally executed models were tested under consistent hardware constraints. The evaluation combined automated RAGAS-based metrics with a manual audit of 40 applications across four formal requirement categories: budget, overhead, applicant role, and letter of intent. The results show that performance varied substantially depending on both the model and the requirement type. The strongest results were observed for overhead and role-based checks, while budget evaluation was consistently more difficult due to errors in both numerical extraction and rule application. The improved RAG pipeline increased retrieval quality, especially context precision, but did not eliminate model-level weaknesses in…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.