ANSPRE: Improving Question-Answering in Large Language Models with Answer-Prefix Generation
Abstract
Large language models (LLMs) and Retrieval-Augmented-Generation (RAG) show remarkable capabilities in Open-domain question-answering (ODQA). Despite the advancements, LLMs tend to generate verbose responses, of which only a small part is the answer phrase. Although the ability to produce the confidence score for the answer is essential when deploying LLMs in high-risk domains, sequence probabilities obtained from LLMs do not correlate well with the probabilities of correctness and thus fail to represent confidence scores. This study introduces Answer-prefix Generation (Anspre) to improve generation quality, allowing the LLMs to output answer phrases and produce highly reliable confidence scores. We guide the model in predicting the answer phrase using an answer prefix and design a ranking score that integrates parametric and non-parametric knowledge. The answer phrases and their corresponding scores enable Anspre to aggregate results from different documents and samplings to boost performance and produce confidence scores highly correlated with correctness. We show that Anspre can be applied to any LLM and present an approach called Self-Anspre to combine Anspre with Self-reflective RAG, a state-of-the-art framework based on reflection tokens. Empirical evaluation on popular ODQA benchmarks shows that Anspre and Self-Anspre significantly improve state-of-the-art LLMs and RAG frameworks. An in-depth analysis shows that confidence scores produced by Anspre are highly…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.