“Fast Conversion From UTF-8 with C++, DFAs, and SSE Intrinsics”
summary
A DFA-based converter that uses three lookup tables and simple C++ code converts UTF-8 to UTF-32 while decoding, not as two steps, and uses SSE2 intrinsics to accelerate ASCII runs. The DFA maps to 9 states, 12 character classes; the full transition table is about 876 bytes (14 cache lines) with a 380-byte smaller variant. Benchmarks show the ASCII-optimized version ≈3–4× faster than iconv and the SSE path often 5–6× faster on tested Intel hardware; code assumes little-endian, aligned buffers and no runtime error checking.
AI-written summary · about
chapters (28)
- 0:00Introduction
- 1:03Outline
- 2:12Code Units
- 3:19Encoding
- 4:25Code Points
- 6:13What is UTF8
- 7:17ASCII byte sequences
- 9:30UTF8 byte sequences
- 11:01Overlong sequences
- 12:15Boundary conditions
- 15:52How does it work
- 16:50Simple DFAs
- 18:10How to recognize UTF8
- 21:47How to write a converter
- 29:14The DFA
- 30:10Basic Conversion Algorithm
- 34:43Benchmark
- 36:36Original Algorithm
- 48:33Benchmarks
- 48:46GCC Benchmarks
- 49:25Input Data
- 50:23Reference Libraries
- 50:50Testing Methodology
- 52:06Linux Results
- 55:45Windows Results
- 58:14Reusing the Library
- 59:09Reporting Errors
- 59:21Future Directions
