Embedding qualitative data in LLM semantic space: A tutorial on conceptualization, measurement, and validation
Abstract
Large language model embeddings give any text coordinates in a semantic space, where the cosine between two coordinates indexes their similarity. Standard qualitative practice cannot quantify such closeness: open-ended answers become nominal variables, which represent only difference under dummy codes. Psychometrics obtains such closeness from responses, as the correlations among a scale's items. Whether the cosine can substitute for the correlation is therefore open. We tested it against established findings in open data, each with an agreed structure: the Big Five, PANAS in two languages, Schwartz's values, Pineo-Porter occupational prestige, and the ISEAR and Kjell narratives. Linguistic structure alone did not reproduce the psychometric structure. The item texts recovered part of it, and none of it where a relation carries a sign. They fall far short of a random split-sample ceiling. Read from the other side, that is a finding about questionnaires: a scale's wording does not carry what its respondents give it. Embeddings supply a semantic layer between raw language and conventional coding, where the material that neither psychometrics nor hand coding reaches belongs. A category label—an occupation, a diagnosis, a brand—becomes coordinates rather than a dummy variable. An open-ended answer becomes coordinates without a coder. Cut into segments, a long answer becomes a path, along which departures, returns, and drifts can be drawn and tested. What any of it means, however,…
Read the paper
Related studies
- Regulation of nucleolus assembly by non-coding RNA polymerase II transcripts
- For a Healthy Histone Code, a Little SUMO in the Tail Keeps the Acetyl Away
- Discovering a binary CTCF code with a little help from BORIS
- Article Commentary: Stabilizing the Code–-Methods to Preserve RNA Prove Their Worth
- Foreword to the Special Issue for the RINTC (The Implicit Seismic Risk of Code-Conforming Structures) Project
- Commentary: Promoting Health and Well-being in Pediatric Primary Care Settings: Using Health and Behavior Codes at Routine Well-child Visits
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.