How to parse a file
summary
A practical ShaderLab parser for Rider/Resharper Unity support builds a PSI concrete syntax tree (including whitespace and comments) so the IDE can do highlighting, folding and refactorings. The pipeline uses lexer generators (Lex/Flex, CSharp/FSharp/Java ports) where lexer generates state tables and parser generators or hand-rolled top‑down recursive descent; bottom‑up uses shift‑reduce. Error recovery mixes panic mode, token insertion/removal and grammar-encoded errors. Nested comments, injected CG blocks and incremental parsing are handled with lexer actions and token stream decorators.
AI-written summary · about
chapters (32)
- 0:00Introduction
- 1:00Why pass a file
- 7:00Naive parser
- 14:12Regular expressions
- 19:14State transitions
- 23:29Syntax analysis
- 29:00Topdown parse
- 32:43Topdown vs bottomup
- 35:10Parser combinators
- 36:08FParsec
- 37:14CParsec
- 39:04Filter
- 41:02Syntax Trees
- 41:46Filter Tokens
- 42:05Mechanical Process
- 42:21Significant Whitespace
- 42:49Insert Tokens
- 44:48Nested Comments
- 45:30Summary
- 45:49Surprise tokens
- 46:45Passing
- 47:29Error Handling
- 47:44Why do we care
- 48:51Error elements
- 49:31How to find an error
- 50:00Panic mode
- 51:51Token insertion
- 52:34Token removal
- 53:04Errors
- 53:46Incremental parsing
- 54:41Composable languages
- 56:26Conclusion
related talks
49mAmazing feats of the Abstract Syntax Tree
01h 00mUse your source code to document your application
32mPassing Strings Using (In)finite-State Machines and Prepack
09mA variant of recursive decent parsing
41mBuilding Parsers in C#: From Theory to Practice. The Uklon Case
30mConcrete Syntax Metapatterns
26mAST + AI = Developer Co-Pilot 2.0
28mThe Nano-parsing Architecture: Sane And Portable Parsing For Perverse Environments