Benchmarking the Impact of Retrieval Augmented Generation on HPC Source Code
Abstract
High Performance Computing (HPC) systems are critical enablers of scientific discovery and industrial innovation. As global challenges such as climate change, biomedical research, and material science continue to demand increasingly complex simulations and analyses, the ability to efficiently develop optimized parallel code has become more important than ever. Frameworks like OpenMP provide a pathway for developers to implement parallelism in shared-memory systems, yet utilizing them effectively requires significant domain knowledge. These challenges are exacerbated by the growing complexity of computational hardware and the steep learning curve associated with HPC toolchains. At the same time, the rapid evolution of Large Language Models (LLMs) has opened new frontiers in code generation, allowing developers to automate routine programming tasks. Despite their strengths, general-purpose LLMs are constrained by outdated training data and lack the nuanced understanding required for domains such as HPC. As part of this project, we collaborated with Ph.D. students from the Janesari Lab to upgrade an exisiting benchmark framework that integrates Retrieval Augmented Generation (RAG) techniques into the code generation pipeline to address these limitations. By retrieving relevant and domain-specific context from external corpora like GitHub repositories and StackOverflow discussions, and encoding them using sophisticated embedding models such as GIST and CodeSearch, we enable LLMs…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.