Finding Needles In Big Data Haystacks....
summary
Describes Conduit, a framework that expresses user-journey patterns as nondeterministic finite state machines (NFAs) built from regex-style patterns (inspired by Ken Thompson’s 1968 approach) and applies those NFAs at scale over billions of events using Apache Spark to locate complex event sequences. Covers design and implementation choices made to make the system usable by data engineers, scientists, and analysts, and discusses adapting the approach for real-time use.
AI-written summary · about
chapters (15)
- 0:00Start
- 5:13Typical solutions
- 8:02Introducing "Conduit"
- 10:56Conduit in Action
- 20:47Guiding Principles
- 22:08Game Plan
- 22:40Expression Types
- 23:13Regular Expr - Syntax Tree
- 24:48"Regular Expression Search Algorithm"- Ken Thompson (1968)
- 25:30FSM States
- 26:16Syntax Tree + Finite State Machine (FSM)
- 29:08Dataframe Partition
- 29:26Non Deterministic FSM in Spark Batch
- 29:54FSM evaluation in Spark Batch
- 31:05References
related talks
30mSMART DATA PIPELINES FOR AI
06mNatural Language Processing and Hadoop
55mSearching for Time Series Patterns at Big Data Scale
21mMeeting complex data load and data preparation challenges
54mUntangling Healthcare with Spark and Dataflow
29mThe Data Journey
15mFrom Air Quality to Aircraft and Automobiles, Data Is Everywhere
25mMoving Data From Graph to the Warehouse, A Configuration Driven Approach