Tractable Control for Autoregressive Language Generation
summary
Proposes GeLaTo: distill GPT-2 Large into a tractable HMM PC to compute exact conditioning probabilities and enforce lexical constraints with 100% constraint satisfiability. Trains an HMM with 4,000 hidden states and 50,000 emissions on 8M GPT-2 samples (40 epochs, ~20 hours), compiles keyword/order constraints to deterministic circuits, and guides autoregressive sampling. Generation latency rises (GPT-2 ~20s vs GeLaTo ~100s; search baselines 700–800s) while achieving CommonGen state-of-the-art on standard metrics.
AI-written summary · about
chapters (9)
- 0:00Introduction to language models
- 4:07The challenge of conditioning
- 9:28Probabilistic circuits and HMMs
- 15:10Encoding logical constraints
- 20:54Recursive circuit construction
- 27:42Generalizing the approach
- 31:15The Gelato framework
- 33:58Experiments and evaluation
- 37:53Discussion and Q&A
related talks
43mState of GPT
01h 01mInside GPT – Large Language Models Demystified
01h 00mInside GPT – Large Language Models Demystified
43mBeyond the Hype: A Realistic Look at Large Language Models
47mInside GPT – Large Language Models Demystified
45mHow Generative AI Works: LLMs and RAG Under the Hood
28mGPT or how I learned to stop worrying and love the generative AI
35mChatGPT is lying, how can we fix it?