PREVIEW: Real-time Stream Processing at Scale

on YouTube ↗06m

" summary

New Relic describes switching to a streaming system built on Kafka to avoid a tenfold scaling nightmare, running 90 brokers across 600 topics and 4,000 partitions and ingesting about 15 million messages/sec. The team moves real-time monitoring events into Kafka as backbone and a datastore, replaces a timer-driven batch job that polled a custom-built database with continuous streams, publishes per-minute query outputs to Kafka topics for alerts and charts, and highlights inefficiencies of repeated historical queries and indexing bottlenecks.

AI-written summary · about

" chapters (7)

1. 0:00Intro
2. 0:28About New Relic
3. 1:27Kafka at New Relic
4. 2:18Database
5. 2:58Events
6. 3:37Aggregate
7. 4:18Summary

" related talks

• 30mEvent Streaming and the Future of ApplicationsBen Stopford · JAX 2019
• 47mBuilding High Performance Real-time Stream Processing MicroservicesFawaz Ghali · GeeCON 2023
• 31mStreaming: Real-Time Architecture for Connected Products in AutomotiveBaran Demir, Dima Fateev · XConf Europe 2025
• 31mEvent Streaming on RailsBrad Urani · RailsConf 2022
• 27mEvent Streaming Patterns for Ruby ServicesBrad Urani · RubyConf TH 2023
• 05mGo for Real Time Streaming ArchitecturesMickaël Rémond · dotGo 2017
• 36mStream All The Things!Dean Wampler · Scale By the Bay 2017
• 01h 03mApache Kafka Event streaming platform for .NET developersViktor Gamov · NDC Sydney 2019