Validating LLMs as Coders of Cultural Constructs: a Large-Scale Test of Minimally Counterintuitiveness using Pokémon
Abstract
Minimally counterintuitiveness (MCI) theory predicts that concepts violating a small number of intuitive ontological expectations (1--2 violations) are transmitted more successfully than either fully intuitive concepts or maximally counterintuitive (3+ violations) ones. Testing this inverted-U prediction at scale has been limited by the cost of human content coding. We evaluate whether large language models (LLMs) can serve as valid, scalable coders of MCI violations, and use a validated pipeline to re-test the inverted-U prediction on a substantially larger sample than prior work. In Study 1, zero-shot coding by two LLMs (Claude and Gemini) against a human-coded baseline showed only moderate agreement driven by systematic over-attribution of violations. In Study 2, a leave-one-out few-shot calibration procedure -- coding each item zero-shot while supplying the other human-coded items as in-context examples -- raised agreement to nearly double specificity while preserving sensitivity. In Study 3, we applied the calibrated pipeline to Generation 2+ Pok\'{e}mon and merged the result with an independent popularity ranking to test the inverted-U prediction. The predicted curvature became significant, with the fitted parabola's vertex at MCI score 2.04, overlapping theory's predicted optimum. Together these studies show that a modest human-coded seed set can calibrate an LLM coder well enough to test a classic cultural-transmission theory at a scale far beyond what human coding…
Read the paper
Related studies
- Regulation of nucleolus assembly by non-coding RNA polymerase II transcripts
- For a Healthy Histone Code, a Little SUMO in the Tail Keeps the Acetyl Away
- Discovering a binary CTCF code with a little help from BORIS
- Article Commentary: Stabilizing the Code–-Methods to Preserve RNA Prove Their Worth
- Foreword to the Special Issue for the RINTC (The Implicit Seismic Risk of Code-Conforming Structures) Project
- Commentary: Promoting Health and Well-being in Pediatric Primary Care Settings: Using Health and Behavior Codes at Routine Well-child Visits
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.