Evaluating Artificial Intelligence-Assisted Current Procedural Terminology Coding in Vascular Surgery: A Comparison of ChatGPT Plus and Perplexity Pro Against Finance Department
Abstract
BACKGROUND: This study evaluates the performance of ChatGPT Plus and Perplexity Pro in matching Current Procedural Terminology (CPT) codes from vascular surgery cases at Tufts Medical Center, comparing their accuracy to that of the finance department's CPT coding, which serves as the reference standard.
METHODS: A total of 120 vascular surgery cases from April 2024 were analyzed. Each case was documented in two formats: operative notes with detailed procedural descriptions and brief operative summaries. Both artificial intelligence (AI) models were tested using these formats, and their CPT code outputs were compared to the finance department's codes. Performance was assessed at two levels: CPT-level accuracy (individual code matching) and case-level accuracy (entire case coding correctness). AI-generated codes were categorized as exact match, partial match, or no match at the case level. Cohen's kappa analysis was then used to measure inter-rater agreement.
RESULTS: ChatGPT and Perplexity Pro produced varying levels of accuracy, with ChatGPT tending to over-report CPT codes, while Perplexity AI under-reported in full-note cases but over-reported in brief notes. Using full operative notes, ChatGPT matched 45.2% of CPT codes, while Perplexity AI matched 43.7%. At the case level, ChatGPT exactly matched 29.2% of cases, partially matched 49.2%, and had no match in 21.6%. Perplexity AI had a higher exact-match rate (39.2%) but also had more cases with no match (27.5%). With brief…
How strong is the evidence?
Evidence Hub reads this as a cohort study, done in people. Its evidence level is 3 of 5 (Moderate · trial / cohort / secondary analysis). Trials without a large randomized design, cohorts and secondary analyses give moderate evidence.
Read the paper
Related studies
- Regulation of nucleolus assembly by non-coding RNA polymerase II transcripts
- For a Healthy Histone Code, a Little SUMO in the Tail Keeps the Acetyl Away
- Discovering a binary CTCF code with a little help from BORIS
- Article Commentary: Stabilizing the Code–-Methods to Preserve RNA Prove Their Worth
- Foreword to the Special Issue for the RINTC (The Implicit Seismic Risk of Code-Conforming Structures) Project
- Commentary: Promoting Health and Well-being in Pediatric Primary Care Settings: Using Health and Behavior Codes at Routine Well-child Visits
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.