Evaluating Large Language Models Trained on Code
Abstract
I created a basic proof of concept of LMVM (Language Model Virtual Machine), a command line toolchain that English instructions into native binaries without intermediary high-level language compilation. The command line uses a large language model (LLM) as IR generation backend, producing LLVM Intermediate Representation (IR) directly from user instructions coding. The IR is compiled and linked to machine code (-o files) via `clang` (with an `llc`/`lld` fallback on POSIX hosts) and executed on bare metal, within Docker sandboxes, on AWS Lambda, or as standalone HTTP servers. How It Works (5 Steps) 1. You type an instruction - e.g. "Create a rest api running on localhost port 3000 and return hello world" 2. LMVM checks your computer - It figures out if you're on Windows/Mac/Linux, what CPU you have (Intel vs ARM), and makes sure clang (the compiler) is installed. If not, it offers to install it for you (with your permission). 3. An AI writes LLVM IR - Instead of writing C or Python, the LLM writes LLVM IR -- a low-level assembly-like language that sits between source code and machine code. The prompt includes your OS details so the AI generates code that works on your platform (e.g. Winsock on Windows vs POSIX sockets on Mac/Linux). 4. Clang turns IR into a native binary - clang compiles the IR and links it with system libraries (PostgreSQL, OpenSSL, SQLite, etc.) into a real executable. One step, no extra tools needed. 5. It runs - The binary executes directly on your CPU.…
Read the paper
Related studies
- Regulation of nucleolus assembly by non-coding RNA polymerase II transcripts
- For a Healthy Histone Code, a Little SUMO in the Tail Keeps the Acetyl Away
- Discovering a binary CTCF code with a little help from BORIS
- Article Commentary: Stabilizing the Code–-Methods to Preserve RNA Prove Their Worth
- Foreword to the Special Issue for the RINTC (The Implicit Seismic Risk of Code-Conforming Structures) Project
- Commentary: Promoting Health and Well-being in Pediatric Primary Care Settings: Using Health and Behavior Codes at Routine Well-child Visits
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.