An Evaluation Study of Generative AI Systems: Framework-Aware Performance Under Real-World Constraints
Abstract
The widespread adoption of Large Language Models (LLMs) in enterprise applications has created a critical need for systematic evaluation of Generative AI Systems (GenAIS) that integrate orchestration frameworks, foundation models, and deployment optimizations. This paper presents the first comprehensive evaluation study specifically designed to evaluate the performance trade-offs between orchestration frameworks (LangChain, LlamaIndex), foundation models, and deployment constraints across multiple application domains. Our study systematically evaluates eight foundation models across question-answering with Retrieval-Augmented Generation (RAG) and mathematical reasoning tasks, measuring latency, accuracy, resource utilization, and power consumption under various optimization strategies including adaptive context windowing and concurrent request processing. Through extensive empirical analysis, we demonstrate that framework selection significantly impacts system performance independent of model choice, with task-specific trade-offs emerging across workload types. For retrieval-heavy RAG workloads, LlamaIndex delivers 3-15% lower latency, around 60% lower peak memory usage, and 2-4% more energy consumption. For reasoning-intensive mathematical tasks, Langchain achieves higher accuracy across all tested models and provides more predictable latency profiles and reasonable resource consumption, making it preferable for strict SLA environments. Adaptive context windowing does not…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.