Code World Model: Building World Models for Computation
summary
A 32 billion parameter dense transformer models program execution by generating execution traces with local variables and predicting state transitions, trained on a few trillion pretrain tokens and ~200 billion post‑training tokens; it runs repo‑level CI to collect traces and learns in a bash‑oriented agentic environment using asynchronous RL with queued samplers and mid‑trajectory checkpoint swaps. The model traces functions line‑by‑line with high accuracy, supports a neural debugger workflow, and approximates halting‑style reasoning; weights are available on Hugging Face.
AI-written summary · about
chapters (8)
- 0:00Introduction to code models
- 1:36Modeling program execution
- 3:33World models and reasoning
- 4:51Data collection for CWM
- 6:06Training the CWM artifact
- 8:43Scaling asynchronous RL
- 11:34Applications and capabilities
- 14:01Future goals and conclusion
related talks
34mBeyond Transformers - Intro to RWKV Architecture & The World To...
36mNeural Networks
25mTraining Large Language Models in the Open
34mDiving into Transformer Model Internals
40mAn Occam's Razor Principle for Transformers?
09mCrypto World Model, From On-Chain Data to Intelligence
25mTransformers.js: Building Next-Generation WebAI Applications
26mRethinking the Transformer: Toward Native Multimodal Architectures