Task-Oriented RAG Evaluation: Raw Generation and Retrieval Results
Abstract
This repository contains the raw experimental outputs produced during evaluation of locally deployed large language model (LLM) and retrieval-augmented generation (RAG) systems. The records capture the outputs of the experimental generation pipeline before downstream evaluation metrics were computed. They include standalone chat-mode generations and RAG-mode generations across multiple model, hardware, platform, embedding, and reranking configurations. For RAG runs, each record preserves the retrieved chunks, the subset selected for use after reranking, the generated response, and system-performance measurements. For standalone chat runs, the same experimental logging structure is retained without retrieval-dependent content. The repository also includes the persistent retrieval databases used by the RAG pipeline. These consist of ChromaDB vector stores and Whoosh lexical indexes corresponding to the evaluated embedding and hardware configurations. --- Relationship to the QAC Benchmark The experiments were conducted using a Question-Answer-Citation (QAC) benchmark organized according to ten information-seeking task categories: 1. Reference / Navigational2. Factoid3. Definition / Conceptual4. Summarization5. Procedural / Task-Oriented6. Policy Interpretation / Compliance7. Comparative / Cross-Document8. Analytical / Rationale9. Ambiguous / Exploratory10. Out-of-Scope / Refusal-Required Each experimental observation corresponds to one question variant drawn from the QAC…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.