A Comparative Evaluation of Retrieval-Augmented Generation For Arabic Documents
Abstract
Generative AI creates new content using AI models (i.e., LLMs) trained on large datasets. However, LLMs lack specific sources and can produce inaccurate or misleading information. Retrieval-augmented generation (RAG) improves accuracy by combining LLMs with a content store, providing evidence for responses and making the model more adaptable to changing information. This paper introduces a comparative evaluation of various generative AI models—LLama 3, LLama 3.1, Gemma, Gemma 2, Phi 3, Phi 3.5, and Phi 3.5 Mini based on their performance in answering questions derived from Arabic and English documents using the RAG locally. The models were evaluated based on their ability to answer questions of varying difficulty levels. Questions were categorized as easy, moderate, or complex. The models’ performance was measured in terms of two key metrics: response time to answer each question and the accuracy of their responses (or quality of the answer). Using RAG method, the results show that LLaMA 3.1 consistently outperformed other models in both Arabic and English languages, whereas Gemma 2 demonstrated decent results. Additionally, response time can be misleading indicators of model performance because less accurate responses require less effort and are generated more rapidly than highly accurate ones.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.