Building and scaling a log analytics platform - a serverless approach

on YouTube ↗32m

" summary

Claims a production grade serverless logging stack built on AWS that replaces Kafka/ELK with Kinesis + Lambda pipeline feeding both S3 + Athena + QuickSight for batch and real-time SQL on streams for live analytics. Demonstrates a working demo: producers push JSON into Kinesis, Lambda transforms, Kinesis Firehose writes to s3, Athena creates tables, QuickSight visualizes and CloudWatch alerts. Notes autoscaling and cost tradeoffs, occasional latency caveats, and that the prototype runs end to end in minutes.

AI-written summary · about

" chapters (13)

1. 0:00Start
2. 0:43A BIT ABOUT ME...
3. 1:53OUR DISTRIBUTED MICROSERVICES
4. 2:24ARCHITECTING A LOGGING FRAMEWORK
5. 2:37BEHAVIOURS OF DIFFERENT MICROSERVICES
6. 3:25EXPECTATIONS FROM ANY LOG FRAMEWORK
7. 4:39THE CONVENTIONAL ARCHITECTURE
8. 7:35WHATS THE COMMON THING THAT ALL THE COMPONENTS HAVE!
9. 8:09GOING SERVERLESS
10. 9:55AWS SERVERLESS
11. 10:18FAAS ARCHITECTURE
12. 13:00MANAGING THE SERVERLESS MODEL
13. 13:41DEMO

" related talks

• 47mBuilding Data Lake Platform Fully ServerlessAnurag Kale · DevOps Pro Europe 2024
• 01h 00mServerless Architecture - Tales from a world without serversRobin Weston · NDC London 2017
• 01h 02mCentralised Logging without the blood, sweat and tearsPaul Stack · NDC Sydney 2016
• 42mBuild and automate a modern serverless data lake onSrinivas Kesanapally, Danielle Greshock · AWS re:Invent 2019 Global Partner Summit
• 12mServerless Stream Processing with Confluent Cloud, Kafka and AWS LambdaGeetha Anne · Moar Serverless 2022
• 01h 02mServerless in production, an experience reportYan Cui, Domas Lasauskas · NDC London 2018
• 39mServerless in production, an experience reportYan Cui · Codemotion Milan 2017
• 49mServerless Architectural Patterns and Best PracticesSascha Möllering · GOTO Berlin 2018