Break the Logs

on YouTube ↗54m

" summary

Describes Alfa-Bank’s logging system built around Apache Kafka, explaining components used to separate log production and consumption. Reviews each component’s limits, how to find bottlenecks in a complex pipeline, and gives resource and load recommendations for collecting logs from large systems.

AI-written summary · about

" chapters (19)

1. 0:00Introduction
2. 1:24Application architecture
3. 4:13Current logging issues
4. 7:22Elasticsearch and Fluentd solution
5. 8:055-year log storage
6. 10:11Log consumer problems
7. 12:14Monitoring with Elasticsearch
8. 15:45Queues and log loss
9. 17:11Choosing Kafka
10. 23:08Kafka performance
11. 25:12Kafka reliability: replication
12. 27:41Kafka reliability: recovery
13. 29:28Security and requirements
14. 31:36Fluentd and Kafka selection
15. 33:00Infrastructure limits
16. 34:25System demo
17. 37:12Real-time log analysis
18. 47:02System benefits summary
19. 48:50Q&A

" related talks

• 56mBrick-by-brick: Exploring Apache Kafka®Danica Fine · Women Techmakers Montreal 2024
• 45mHow LinkedIn is leveraging kafka for log aggregation and monitoringMuzammil Rehman · Rootconf 2015
• 13mKeynote – Jay KrepsJay Kreps · PGConf Silicon Valley 2015
• 29mApache Kafka and the Next 700 Stream Processing SystemsJay Kreps · Strange Loop 2015
• 41mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia November 2020
• 44mHow to Manage State in Apache KafkaBen Abramson · JAX London 2019
• 02h 05mA Deep Dive into Apache Kafka This is Event StreamingKatherine Stanley, Andrew Dunnings · Devoxx Belgium 2019
• 39mKafka as a Platform: The Ecosystem from the Ground UpRobin Moffatt · GOTOpia Europe 2020