Events Analysis with Riak, Perl and Erlang at Booking.com
summary
Booking.com stores >50k events/sec, keeps 8–10 days hot and ships older data to Hadoop, aggregates per-second event blobs serialized with Sereal serialization and stored in Riak using the BigCask backend, initially on 12-node clusters grown to 32+ nodes. Real-time crunching moves to node-local execution via a post-commit hook calling a companion REST service to avoid repeating decompression; MapReduce on-node hits CPU duplication. Additional wins: reduce cross-node traffic with client-side ring hashing and n_val=1 reads.
AI-written summary · about
chapters (30)
- 0:00Introduction
- 3:21What is an Even
- 5:41Even Flow
- 6:32Graphs
- 7:20Events Flow
- 8:07Short Term Analysis
- 8:36Anomaly Detector
- 9:09Data Driven
- 9:33Overview
- 10:29Aggregate
- 12:02Logger
- 14:39React
- 16:25Riak Backend
- 19:00Data Design
- 20:03Data Keys
- 21:41Data Architecture
- 22:06Pseudocode
- 22:59Beating Data
- 26:26Realtime processing
- 28:36Network saturation
- 30:27MapReduce jobs
- 32:01Commit hook
- 34:57Red service
- 36:39Disadvantages
- 37:20The bandwidth problem
- 37:55Fetching data from Riak
- 38:47Cluster usage
- 40:21Fetching from a primary node
- 42:59Results
- 44:58Conclusion
related talks
28mAn introduction to Riak
38mHow to handle 100k events per second
39mUsing Erlang, Riak and the ORSWOT CRDT at bet365 (...)
40mOpen Riak how we gained from leaning on the BEAM
47mSpeeding Event Processing to 1.6M/sec: Badoo’s Experience
43mRealtime Analytics with StarRocks in Wellness Booking
45mRiak Pipe, Distributed Data Processing
46mScalable Realtime Analytics using Druid