Logging Apache Spark How we made it easy

on YouTube ↗13m

" summary

Centralized log search for Spark on EMR uses Filebeat on every EC2 node and Metricbeat, shipping lines to Redis -> Logstash -> Elasticsearch -> Kibana via bootstrap bash scripts that install RPMs edit YAML with sed and wait for HDFS and JDK. Logs get tagged with team name flow name and EMR cluster id so queries are precise. The system surfaces over 30,000 warnings and 23 errors in dashboards, works in production, and later replaces Redis with Kafka to remove a bottleneck.

AI-written summary · about

" chapters (9)

1. 0:00Start
2. 1:08Nielsen's Data Architecture overview
3. 2:06How we used to do it
4. 4:51Solution architecture
5. 5:25Solution overview
6. 6:02Bootstrapping
7. 9:38YAML Template File
8. 10:08Installing Metricbeat
9. 12:00Future and add-ons

" related talks

• 16mLogging Apache Spark - How we made it easySimona Meriam · Berlin Buzzwords 2022
• 45mInteractively Analyse 100GB of DataRaoul-Gabriel Urma, Valentin Dalibard · PyData London 2017
• 14mCentralized Logs using ELK (Elasticsearch, Logstash, Kibana) StackEstu Fardani · FOSSASIA Summit 2017
• 44mRiak, Redis, Apache Solr & Spark: Deploying with the Basho Data PlatformBryan Hunt · GOTO London 2015
• 32mScalable Monitoring Using Prometheus with Apache SparkDiane Feddema, Zak Hassan · KubeCon + CloudNativeCon Europe 2018
• 11mMedic for Apache Spark - First Aid for Failing JobsDrasko Profirovic · AI Engineer World's Fair 2026
• 26mFrom Spark to Elasticsearch and BackSonya Liberman · Big Data Conference Europe 2019
• 24mCICD Pipeline and delivery of Apache Spark Applications on the cloud using AWSMaria Fung, Babu Repaka · Data Con LA 2020