Taming the BEAST : Assessing Open-Source LLM Capabilities in a Local Retrieval-Augmented Generation Assistant
Abstract
Evaluating agentic retrieval-augmented generation (RAG) systems in .NET is constrained by a lack of native tooling. Existing libraries either require a specific language model framework or lack important metrics. This thesis addresses that gap by developing and evaluating BEAST (BEA Stöd), an AI assistant for employees at Uppsala Kommun covered by the BEA collective agreement, together with a custom evaluation framework written in its .NET 9 codebase. The framework covers retrieval, faithfulness, tool usage, abstention, multi-turn dialogue, and inference metrics. Four open-source models are evaluated on Swedish administrative text: Gemma 4 E4B, Llama 3.1 8B, GPT-OSS 20B, and Mistral Small 3.2 24B. The results show that parameter count is a poor predictor of suitability in this domain. Gemma 4 E4B achieves the highest answer relevance and faithfulness despite being the smallest model, while Mistral Small 3.2 24B leads on tool usage reliability but is too slow for practical deployment. Llama 3.1 8B is the fastest model by a large margin but fails to refuse out-of-scope requests in nearly every test case. Context precision is low across all models, pointing to poor retrieval mechanics. The findings demonstrate that .NET native evaluation coverage is achievable without external tools and provides a reusable pattern for teams developing similar systems. Empirically derived thresholds offer a starting point for regression detection in continuously developed RAG assistant pipelines.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.