Fine grained reranking via caption bridging for knowledge augmented visual question answering
Abstract
Retrieval-Augmented Generation (RAG) has emerged as a pivotal framework for knowledge-intensive reasoning by coupling external retrieval with generative capabilities. However, existing RAG systems suffer from a critical granularity mismatch problem: coarse-grained retrieval units (entire passages or global images) fail to align with the fine-grained reasoning requirements of generation tasks, particularly in multimodal contexts where localized visual evidence is essential. We propose Fine-Grained Retrieval-Augmented Generation (FG-RAG), a unified framework that bridges this semantic resolution gap through two key innovations. First, we enhance the CLIP architecture with patch-level contrastive supervision, enabling explicit alignment between localized image regions and corresponding textual fragments. Second, we introduce a joint retrieval-reranking optimization mechanism that unifies a dense retriever with a large language model (LLM)-based reranker through a shared relevance loss. To address the non-differentiable nature of LLM generation, we employ a Score Alignment Strategy where generative likelihoods provide structural supervision for the retriever, creating bidirectional feedback between retrieval precision and generation quality. Comprehensive evaluation on MSCOCO and Flickr30k benchmarks demonstrates that FG-RAG achieves significant retrieval gains (Recall@1 = 0.8845 on MSCOCO), outperforming state-of-the-art methods by up to 7.5% across datasets. In visual question…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.