The use of Guided Generation to Enhance Extraction of Information from Technical Research Papers
Abstract
This project studied whether "guided generation"-forcing a Large Language Model (LLM) to answer a question using a structured format-helps extract information from technical research literature better than letting it answer free form.Finding answers in technical literature has historically meant keyword search plus manual review; LLMs are changing that, and Retrieval-Augmented Generation (RAG), which grounds an LLM's answers in retrieved source passages, is an alternative that is explored in the AI community 1 .Using a corpus built from the International Symposium for Testing and Failure Analysis (ISTFA) proceedings 2 , we generated answers under ten conditions-a no-retrieval baseline, two synthetic degraded conditions for validating the judge, five RAG retrieval conditions, the guided generation condition, and "pondering," a two-turn condition.We tested these with two completion models (Mistral-Small 24B and GPT-OSS 120B) and two embedding backends (a local Model2Vec embedder and Google's EmbeddingGemma).All answers (~431 per configuration) were scored 0-10 by a fixed LLM-as-judge, which we first validated: it ranks deliberately degraded answers monotonically (No-RAG baseline > Mostly Right > False), confirming it scores correctness rather than fluency.Retrieval reliably improved quality over the no-RAG baseline, with most gains within two to three documents.Critically, guided generation did not materially help and actively harmed the smaller model (-1.9 to -2.3 points…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.