Automotive Cybersecurity QA and RAG Evaluation Dataset
Abstract
Automotive Cybersecurity QA and RAG Evaluation Dataset This dataset provides a collection of human-curated question–answer pairs in the automotive cybersecurity domain, together with model-generated answers obtained using multiple retrieval-augmented generation (RAG) configurations. It is designed to support research on domain-specific question answering, retrieval-augmented generation, and the evaluation of large language models in automotive cybersecurity. Additional details on the RAG configurations, retrieval strategies, and evaluation methodology are described in the associated publication. The dataset includes: A set of human-written question–answer pairs created by domain experts, serving as ground truth. Model-generated answers produced under four different RAG configurations: ensemble, semantic, syntactic, and two-stage. Dataset Structure human_answers/: Contains human-written reference answers. answers_rag/: Contains model-generated answers, organized by RAG configuration (ensemble, semantic, syntactic, two_stage). Each folder contains CSV files corresponding to different evaluated language models. Data Format All files follow the same schema: q_id: Unique identifier of the question question: The natural language question answer: The corresponding answer (human or model-generated depending on the file) Purpose This dataset can be used to: Benchmark large language models on domain-specific question answering Evaluate and compare different RAG strategies Analyze and…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.