Big Data

on YouTube ↗19m

" summary

CERN pipelines petabyte-scale experiment and infrastructure logs into a stack built on Apache Kafka for buffering, HDFS/Hadoop for long-term storage and Spark Streaming for real-time analytics. A distance-based anomaly detector runs on Spark Streaming and emits alerts live; Elastic keeps short-term indexes (≈13 days) and notebooks/Impala enable ad hoc queries. A distributed profiler aggregates per-machine CPU and I/O metrics to find hotspots across clusters, enabling migration from Oracle to scalable stores and measurable performance improvements in production.

AI-written summary · about

" chapters (9)

1. 0:00Big Data introduction
2. 2:13Programmatic data access
3. 4:04Sensors and actuators integration
4. 6:13Distributed storage and processing
5. 8:22Centralized monitoring and real-time logs
6. 10:30Streaming processing and anomaly detection
7. 13:00Network distance learning
8. 16:15Perfilado de hardware en aplicaciones distribuidas
9. 18:46Summary and Q&A

" related talks

• 52mDevil in the HaystackPing Yan · AppSec California 2015
• 27mBeyond Logs: Real Time Anomaly Detection with Machine LearningFawaz Ghali · Great International Developer Summit 2023
• 37mBig Data Analytics, Machine Learning, Real Time ProcessingKai Waehner · Codemotion Milan 2016
• 01h 35mDeep Dive into the Big Data LandscapeDean Wampler, Eva Andreasson · GOTO Copenhagen 2014
• 50mData Science in the EnterpriseBas Geerdink, Natalino Busa · J-Fall 2015
• 43mBeyond Logs: Real-Time Anomaly Detection with Machine LearningFawaz Ghali · Voxxed Days Thessaloniki 2023
• 44mAnomaly Detection in Telecom with SparkTugdual Grall · Codemotion Amsterdam 2016
• 30mAlice and Mad Hatter....Marianna Diachuk, Roksolana Diachuk · Scale By the Bay 2021