Benchmarking the Impact of Contextual Information on OpenMP Code Generation by Large Language Models
Abstract
The rapid development of tools based on artificial intelligence has been driven by the widespread adoption of large language models (LLMs). These models have opened up new opportunities for automating tasks that were previously considered manual and labor-intensive, enabling end-to-end automation. For example, LLMs have been applied in various industries to power chatbots that assist customers by answering questions about products. A key technique utilized in many of these applications is retrieval augmented generation (RAG). Existing benchmarks for evaluating the performance of LLMs in RAG have typically focused on documents from Wikipedia and news websites. In this study, we extend the concept of RAG to retrieval augmented code generation (RACG), aimed at helping LLMs generate domain-specific OpenMP source code. Code generation in this context is particularly challenging due to the need for precise handling of function signatures, method calls, and other domain-specific constructs. We use contextual information from Stackoverflow posts and GitHub repositories to aid in this task. To assess code generation performance, we evaluated 15 LLMs of varying sizes and computed the CodeBLEU and CodeBERT metric. Our findings show that (1) most LLMs can generate OpenMP code effectively without context, (2) GitHub code snippets provide more useful context for LLMs than Stackoverflow posts, (3) context helps smaller and non-code-focused models improve their ability to generate OpenMP…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.