Design and evaluation of a Retrieval-Augmented Generation system for Maxtech Support Portal : Comparing Vertex AI, Qdrant, and Pinecone architectures in a Finnish-language enterprise context
Abstract
This thesis investigates the creation, implementation and assessment of a Retrieval Augmented Generation (RAG) system specifically designed for the Maxtech Support Portal. The work addresses a real-world enterprise problem whereby users frequently open support tickets that are already documented and addressed on the portal, but users are still struggling to locate relevant information independently. Three different RAG systems were conceptualized and subsequently comparatively tested against each other, one based on the Google Vertex AI integrated solution and two fully custom developed solutions that were tested using two well - established cloud vector databases: Qdrant Cloud and Pinecone. The systems were evaluated by focusing on the component-level evaluation: metadata - filtering, reranking of results, question augmentation and enrichment of documentation with image data at ingestion time, all contributing to answer generation. The RAG Evaluation Dashboard was created to support controlled blind evaluation, structured human scoring, and descriptive analysis. A total of 30 real support questions were picked to go through human assessment, making a total of 270 ratings over three architectures and three evaluators. Responses were rated based on five quality dimensions: accuracy, completeness, faithfulness, relevance, and usefulness. The results show that each of the three architectures was technically feasible to support enterprise automation but with distinctly different…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.