Logging Apache Spark - How we made it easy

on YouTube ↗16m

" summary

Nielsen runs Spark on Amazon EMR processing more than 60 terabytes and billions of events daily and the team builds a bootstrap bash script that installs Filebeat and Metricbeat on non‑master EC2 nodes, parses diverse legacy log4j formats with a multi‑line regex, waits for HDFS, ships to Redis → Logstash → Elastic and surfaces actionable Kibana dashboards. The system makes live searching far faster than SSH or S3 pulls, exposes EMR cluster id and Spark component per log line, and later replaces Redis with Kafka to remove a bottleneck.

AI-written summary · about

" chapters (10)

1. 0:00Start
2. 2:36Nielsen's Data Architecture overview
3. 3:45How we used to do it
4. 6:31Solution architecture
5. 7:07Solution overview
6. 8:32Installing Filebeat
7. 12:04YAML Template File
8. 12:48Main Obstacles
9. 13:18Pretty Kibana Charts
10. 13:58Future and add-ons

" related talks

• 13mLogging Apache Spark How we made it easySimona Meriam · NDC Oslo 2021
• 45mInteractively Analyse 100GB of DataRaoul-Gabriel Urma, Valentin Dalibard · PyData London 2017
• 44mRiak, Redis, Apache Solr & Spark: Deploying with the Basho Data PlatformBryan Hunt · GOTO London 2015
• 34mKick-Ass Tools and Processes for Scalable Data ScienceHelge Reikerås · ScaleConf 2019
• 35mScaling up to Big Data Devops for Data ScienceMarck Vaisman · PyData DC 2016
• 24mCICD Pipeline and delivery of Apache Spark Applications on the cloud using AWSMaria Fung, Babu Repaka · Data Con LA 2020
• 44mAutomating a Big Data Platform with JenkinsSander Kieft · Jenkins User Conference U.S. East 2015
• 32mBuilding Analytics Workflow using Airflow and SparkYohei Onishi · PyCon APAC 2019