From Prompt to Pipeline: A Comparative Evaluation of Large Language Model Coding Agents for Reproducible Bioinformatics Pipeline Construction
Abstract
Agentic coding systems are increasingly presented as a way to reduce the engineering burden of scientific software development. Bioinformatics is a strong test case for this claim because useful pipelines must combine domain-specific analysis choices, command-line software, sample metadata, workflow orchestration, container or HPC execution, and interpretable quality-control reporting. We evaluated three agentic systems - Biomni, Claude Code, and Codex - on the same task: constructing a Nextflow DSL2 pipeline for paired-end CUT&Tag data that included read QC, trimming, alignment, filtering, duplicate removal, signal track generation, per-sample and group-level peak calling, control-aware group merging, annotation, FRiP calculation, deepTools visualizations, and final MultiQC reporting. Each system received the same detailed CRAFT-style prompt and was assessed against a hand-coded reference pipeline developed by the authors. All three systems produced pipeline implementations that appeared plausible at the level of documentation and file structure, but none fully satisfied the requested analysis. The most consequential failure was shared: the agent-generated pipelines performed some form of group-level merging but did not produce the requested merged-group reporting outputs. Sample-level MultiQC reports also disagreed with the reference report. Codex was closest to the reference for primary mapped-read counts, although its total-read accounting and report structure still…
How strong is the evidence?
Evidence Hub reads this as a preprint, done in computer models (in silico), not in people. Its evidence level is 5 of 5 (Very low · case / narrative / animal / in vitro / in silico). Case reports, narrative reviews, and animal, cell or computer-model studies give the weakest evidence for human health.
Read the paper
Related studies
- Regulation of nucleolus assembly by non-coding RNA polymerase II transcripts
- For a Healthy Histone Code, a Little SUMO in the Tail Keeps the Acetyl Away
- Discovering a binary CTCF code with a little help from BORIS
- Article Commentary: Stabilizing the Code–-Methods to Preserve RNA Prove Their Worth
- Foreword to the Special Issue for the RINTC (The Implicit Seismic Risk of Code-Conforming Structures) Project
- Commentary: Promoting Health and Well-being in Pediatric Primary Care Settings: Using Health and Behavior Codes at Routine Well-child Visits
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.