Seeing vs reading: evaluating visual and textual approaches in retrieval-augmented generation
Abstract
This thesis presents a comparative analysis of visual and textual retrieval methods within Retrieval-Augmented Generation (RAG) systems, based on the hypothesis that visual models outperform textual ones in scenarios where visual resources are predominant. To validate this hypothesis, a rigorous experimentation process is conducted, starting with real data provided by a client, ensuring that the evaluated content was not used in any model training, thus offering a realistic evaluation context. As the need for broader experimentation arises, the study expands to include additional datasets such as TAT-DQA and NewsQA, and later the newly released Vidore V2 datasets-ensuring that these were not used in the training phase either. In total, ten datasets are analyzed using seven visual models and three textual models, with all models answering queries using the same context: images extracted through a visual processing pipeline and text obtained via Nuclia's processing system. Throughout the project, fairness in the comparison is strictly maintained, ensuring that the generative model receives equivalent contextual information in both modalities. The results show that there is no universally superior approach: performance depends on the type of query and the nature of the documents. In cases involving rich visual content (such as tables, graphs, or complex layouts), visual models generally perform better by leveraging this additional information. However, this advantage comes at a…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.