"DeepVis Grammar of Ornament Multimodal Document Understanding Benchmark: Stratified Queries and Ground Truth Annotations"
Abstract
"This dataset supports the evaluation of multimodal document understanding systems against a retrieval-augmented generation baseline on a 612-page visually rich historical corpus, Owen Jones's The Grammar of Ornament (1856). It contains 15 stratified evaluation queries across three categories \u2014 text-based (T), visual (V), and cross-modal (M) \u2014 each accompanied by annotated ground truth establishing correct page citations and expected content answers established prior to system evaluation. Full system responses from DeepVis and a RAG baseline using Azure AI Search with GPT-4.1 are included for all 15 queries. The dataset supports four evaluation metrics: Query Success Rate (QSR), Page Citation Precision (PCP), Page Citation Recall (PCR), and Visual Content Accuracy (VCA). It is intended for researchers evaluating vision-language systems, retrieval architectures, and citation accuracy on document collections where meaning depends on visual and spatial content rather than text alone."
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.