Super Fast Real-time Data Processing on Cloud-Native Architecture
summary
Claims an in-production platform that hits 100Gb/s per node and ~2 million transactions/sec per server with 99th-percentile latency ~100 µs by bypassing filesystems and using zero-copy shared memory, NVMe, RDMA/DPDK and NIC offloads. Architecture uses nginx front-ends, a custom back-end key-value engine, an asynchronous FUSE-like daemon for volumes, high-performance RPC and captain proto serialization, Kubernetes for orchestration but keeps some containers outside K8s for features that K8s lacks; components are being open-sourced.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:22Architecture
- 3:32Data Path
- 5:22Data Path Design
- 12:35Challenges
- 14:30Networking in Containers
- 17:34Block Storage
- 23:54Service
- 27:18Data Sources
- 30:05Data Binding
- 30:55Data Integration
- 33:32Example
related talks
43mRealtime performance at scale (...)
18mKeynote: Mind the Gap: Bridging Cloud Native Innovation with Real-World Use
04mCeph and NVMe/TCP
08mReal-Time Data Systems: Speed, Scale, and Resilience
38mExploring Distributed Caching for Faster GPU Training with NVMe, GDS, and RDMA
30mBeyond the DC Walls: Building a Nationwide GPU Fabric With KubeVirt and Wide-Area…
30mBlazing fast I/O of data in the cloud with Daft Dataframes
20mBuilding realtime efficient queries in an IOT platform