Funnel Rocket: A Serverless Query Engine
summary
An open-source engine that runs serverless big-data funnel and sequence queries by composing AWS Lambda, Pandas + Apache Arrow and Parquet on cloud storage, with Spark only for offline prep. Redis acts as the coordination plane, enabling thousands of short-lived workers (default concurrency ~1000) to process per-user histories across millions of users and hundreds of millions of events. Minimal ETL, JSON query API, low orchestration code and real deployments show the approach is practical and cost-effective.
AI-written summary · about
chapters (10)
- 0:00Introduction and speaker background
- 2:45Problem domain at Dynamic Yield
- 10:10Why another query engine
- 14:04Challenges with sequences and funnels
- 17:15Initial approach with Elasticsearch
- 20:47Limitations of cluster-based systems
- 23:14Designing the serverless solution
- 26:47Redis for orchestration
- 29:16How Funnel Rocket works
- 31:42Open source and future steps
related talks
34mDesigning Data Intensive Applications In Serverless Architecture
25mA simple serverless data pipeline
41mUnlocking Faster & Efficient Data Processing w/ Serverless
42mNo Server Is Better Than No Server: Building Blocks for Serverless Architectures
49mUsing serverless functions for real-time observability
25mData Gone in 60 Seconds: A Serverless ETL Heist
47mAdvanced Serverless Architectural Patterns on AWS
41mUsing Serverless & ARM64 for Real-Time Observability