A Three-Stage LLM Evaluation Framework for RAG Systems in Power Marketing Scenarios
Abstract
Retrieval-Augmented Generation (RAG) systems powered by large language models (LLMs) are widely deployed in power marketing services to automate response generation, content review, and user interaction. However, conventional evaluation approaches mainly rely on single-LLM evaluation, which suffers from unstable judgment, boundary sample oscillation, and lack of business-oriented security indicators. To address these limitations, this paper proposes a three-stage automatic evaluation framework for power marketing RAG systems. The framework consists of an LLM primary judgment (Judge), review correction (Review), and conflict arbitration (Arbiter) stage. Under strict prompt engineering constraints, the framework supports comprehensive evaluation over correctness, faithfulness, completeness, hallucination and refusal behavior. We conduct experiments on a real-world industrial dataset with 2,389 samples. Experiments show that, compared with a single-LLM evaluator, our method produces scores closer to human experts and remains stable across repeated runs with low variance. This work provides a reproducible and industry-ready evaluation solution for practical RAG systems.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.