Local Credit Assignment for CPU Transformers: Readout Consensus and the Cost of Predictive Coding
Abstract
Abstract Local objectives expose parallelism across Transformer depth but remove downstream credit. We study this trade-off in two complementary experiments on central processing units (CPUs). A prospectively specified campaign on 24-layer byte-level Transformers evaluates readout-gradient consensus (RGC), which coordinates shared-readout parameter gradients without restoring cross-stage activation gradients. Asynchronous RGC reaches 1.382 times backpropagation throughput, while process-tree memory rises from 1.94 to 4.31 GiB and active worker threads increase from 24 to 29. The selected broadcast method fails its held-out 1\% non-inferiority criterion; TinyStories transfer and the primary readout-equivalence criterion also fail. A separate, exploratory 27-run study applies stage-boundary augmented Lagrangian predictive coding (PC-ALM) to a smaller 12-layer Transformer. Three inference rounds produce a mean paired validation-loss gap of +0.142\% versus backpropagation at 2.78 times its serial training time. Initialising multipliers from local readout gradients degrades learning; penalty-only predictive coding retains much of the quality benefit. We provide a readout-information counterexample, a feasible-stationary-point gradient identity, and a linear conditioning diagnostic, with explicit limits on their implications. The evidence identifies two distinct costs: quality lost by truncating credit and computation spent recovering it. It establishes neither quality-preserving…
Read the paper
Related studies
- Regulation of nucleolus assembly by non-coding RNA polymerase II transcripts
- For a Healthy Histone Code, a Little SUMO in the Tail Keeps the Acetyl Away
- Discovering a binary CTCF code with a little help from BORIS
- Article Commentary: Stabilizing the Code–-Methods to Preserve RNA Prove Their Worth
- Foreword to the Special Issue for the RINTC (The Implicit Seismic Risk of Code-Conforming Structures) Project
- Commentary: Promoting Health and Well-being in Pediatric Primary Care Settings: Using Health and Behavior Codes at Routine Well-child Visits
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.