RAViG-Bench: A Benchmark for Retrieval-Augmented Visually-Rich Generation with Multi-Modal Automated Evaluation
Abstract
Retrieval-Augmented Visually-rich Generation (RAViG) extends RAG by integrating textual explanations with multiple visual elements in a well-structured layout. Despite its growing adoption, no existing benchmark offers a holistic evaluation of RAViG. Current RAG benchmarks focus on text-only generation, while natural language to visualization (NL2VIS) benchmarks focus on ''show-data-as-chart'' style queries and do not follow the RAG paradigm. To address this deficiency, we present RAViG-Bench, the first comprehensive benchmark specifically designed for RAViG. The benchmark features a diverse collection of authentic user queries, each paired with real-world web retrievals to simulate realistic RAViG scenarios. Besides, we introduce a novel multi-modal automated evaluation framework that holistically assesses the quality of RAViG outputs. This framework scrutinizes the generated content by evaluating the functionality, design quality, and content quality of both textual and visual components. Our extensive experiments on leading commercial and open-source LLMs provide a comprehensive analysis of their current capabilities, highlighting significant limitations and charting key directions for future research in this emergent area. The dataset, code, evaluation prompts, and documentation are available at https://github.com/antgroup/ravig-bench.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.