Dropping Doesn’t Mean Losing
summary
Outbrain runs on Kubernetes and handles about 5 million requests/sec, 200M metrics/min and ~15 TB of logs/day, so a lightweight rate-control layer opens incoming Kafka messages, counts combinations of predefined fields, and enforces per-label per-second limits. Initially messages were queued and throttled into Elastic, later the system shifts to selective dropping where only messages above the limit are deleted, while developers get alerts; this reduces cost, keeps Elastic stable, and preserves useful logs for debugging.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 1:49Document types and blogs
- 5:20Data democratization
- 8:54How the bassist works
- 12:49From delay to message removal
related talks
31mLogging at Scale, Billion+ messages, 100K msg/sec
25mPersistent and performant ingestion using Data Prepper
38mData Pipeline with Apache Kafka
32mShould you read Kafka as a stream or in batch? Should you even care?
32mScaling your Kafka streaming pipeline....
38mKafka Message Broker Under High Load
37mScaling Yelp’s Logging Pipeline with Apache Kafka #bbuzz
01mScale By the Bay 2021 intro video by Opher Dubrovsky