Construction and optimization of agentic retrieval augmented generation pipeline and application for knowledge and operation of satellite systems
Abstract
This thesis explores two settings where large language models (LLMs) can be useful for military contexts: (1) the development of an application for learning course material in a graduate-level Space Systems course at the Naval Postgraduate School, SS3861 Spacecraft Payload Design, and (2) assisting new users of military Satellites navigate Satellite Operations (SATOPS) manuals and instructions. Students often turn to LLMs to speed up learning, but with generalized LLM applications, students may learn from models incapable of producing relevant and accurate outputs for these subjects. We show that when utilizing an optimized agentic retrieval augmented generation (RAG) pipeline with a tested open-source LLM, we can build an application capable of outperforming general LLM applications. When creating the RAG pipeline, we began by ingesting documents from both SS3861 and the SATOPS program. We built tools to behave as RAG optimization techniques and incorporated long-term memory using an SQLite database. We tested three open-source models integrated into the RAG pipeline, along with ChatGPT and Claude, which were not integrated into the pipeline. Working with subject matter experts in the relevant domains, we developed a methodology to evaluate the performance of the models in four areas: correctness, reasoning, pedagogy, and citation strength. We found that NVIDIA's Nemotron model outperformed all other models, showing that RAG is crucial for the best tutor performance.
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.