Retrieval-Augmented Generation with Low-Latency Deployment for Vertical Domains Question Answering: A Case Study on Economic Resource Platforms
Abstract
Abstract Generative chatbots have improved access to public information, yet their deployment in vertical domains such as policy documents and economic resource platforms still faces challenges in retrieval accuracy, answer faithfulness, latency control, and scalable service readiness. This paper presents an optimized domain-specific question-answering framework that combines normalized Dense Retrieval and BM25 fusion, DistilBERT-based candidate re-ranking, and evidence-constrained T5-small generation within a retrieval-augmented generation pipeline. DistilBERT is used as an encoder and relevance scorer rather than a generator, providing document weights for downstream answer generation. To support low-latency deployment, the system adopts ONNX Runtime, FP32/FP16/INT8 precision evaluation, dynamic INT8 quantization, a Rust-based asynchronous serving layer, Docker containerization, and Google Kubernetes Engine auto-scaling. Experiments on the ESG website of the City University of New York show that the hybrid retriever improves Recall@5 over pure semantic search, while constrained generation reduces hallucination and keeps F1 above 85%. INT8 deployment lowers average latency to 48 ms with limited F1 loss, and cold-start testing identifies model loading as the main scale-out delay. CONCEPTS Information systems~Information retrieval~Retrieval models and ranking~Combination, fusion and federated search
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.