An Extension to BigCodeBench for Task Characterization and LLM-based Agent Evaluation
Abstract
This upload contains two complementary datasets derived from "BigCodeBench". The first dataset focuses on task characterization, while the second contains information about LLM agents, their responses to the coding tasks, and the corresponding LLM-as-a-judge evaluations. Task characterization dataset The task characterization dataset extends BigCodeBench by categorizing Python coding tasks by domain, layer and complexity, to support research on task allocation. The original information has been preserved, except for the three columns: “test,” “entry_point,” and “doc_struct”; which were removed because they are not required for our work.This version only has information about the coding tasks. It can therefore be used by other researchers to develop approaches to task analysis and comparison, difficulty estimation, task categorization, and classification by skill or domain.The categorization was done by an LLM (model: OpenAI GPT-4o) using the following prompt:"You are a software engineering task classifier.Your job is to analyze a Python development task description and classify it according to several categories.Return the classification in JSON format.Categories:1. layerPossible values:- backend- frontend- data- infrastructure- integration- library 2. domainPossible values:- data_science- machine_learning- security- cryptography- natural_language_processing- optimization- networking- file_processing- image_processing- simulation- automation- web_scraping- database- api-…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.