Liberating Ruby's Parser from Lexer Hacks
summary
Claim: replacing ad hoc lexer flags with a parser-aware scanner (PSR) exposes the real causes of Ruby syntax confusion and lets the lexer stop guessing. The implemention builds a pseudo-scan FSA plus a parser-state accept table, adds functions like accept_token and where_state to follow empty reductions or inspect the parser stack, and performs state-splitting to undo harmful LR-state merges. Results: it explains six layers of ambiguity, removes most of the 13-bit / ~100-site lexer hacks, and makes lexer decisions deterministic.
AI-written summary · about
chapters (14)
- 0:00Introduction to talk
- 1:49Speaker introduction
- 6:26Traditional LR parser limitations
- 8:52PSR scanner parser relationship
- 11:44Layer one parser state answers
- 13:29Layer two empty reductions
- 15:15Layer three parser stack lookup
- 16:39Layer four token boundaries
- 20:14Layer five state merging issues
- 22:25Layer six semantic ambiguity
- 23:50Six layers ambiguity map
- 24:11Practical results of PSR
- 25:14Three key takeaways
- 26:00Q&A and closing
related talks
32mParsing Ruby
32mTime flies like an arrow; Fruit flies like a banana: Parsers for Great Good
31mTime flies like an arrow, fruit flies like a banana
29mHow does Lrama make the Ruby parser grammar G.O.A.T.?
46mGrammar a BNF like Ruby DSL Parsing
54mHacking parse.y
52mA bird's eye view of a parser
27mGenerate Parsers! Prevent Exploits!