Measuring the Fairness Gap Between Retrieval and Generation in RAG Systems using a Cognitive Complexity Framework
Abstract
In this paper, we investigate the problem of quantifying fairness in Retrieval-Augmented Generation (RAG) systems, particularly for complex cognitive tasks that go beyond factual question-answering. While RAG systems have demonstrated effectiveness in information extraction tasks, their fairness implications for cognitively complex tasks - including ideation, content creation, and analytical reasoning - remain under-explored. We propose a novel evaluation framework that extends IR fairness metrics by incorporating centrality-based measures to account for influence of retrieved documents on generated output beyond ranking. Our framework evaluates RAG systems across various cognitive dimensions using two ranking approaches: lexical (BM25) and dense (BGE), and language models of varying sizes. Our findings provide insights into: (1) the propagation of fairness disparities from retrieval to generation phases, and (2) the variation in system performance across different cognitive dimensions.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.