Logging Apache Spark - How we made it easy
summary
Nielsen runs >50 TB and billions of events through Spark on EMR and needed real-time log access instead of SSH, YARN UI or delayed S3 dumps. The team installs filebeat and metricbeat via EMR bootstrap actions on every EC2 node, tags logs with flow/team/EMR_cluster_id, ships to Redis then Logstash → Elasticsearch and visualizes in Kibana; it cuts lookup time dramatically, engineers adoption is the main hurdle, and plans replace Redis with Kafka for scale.
AI-written summary · about
chapters (9)
- 0:00Intro
- 1:13Spark
- 2:22Accessing Spark Logs
- 5:47Bootstrap Actions
- 6:26Install Filebeat
- 10:55Obstacles
- 11:37Charts
- 12:42Future
- 13:41Outro
related talks
32mMove to managed analytics
52mSolving Analytical Problems using Apache Spark
26mRe-architecting the Data Platform with Spark on Kubernetes
38mReal-time analytics with Apache Cassandra & Apache Spark
41mUnifying Diverse Logs in Big Data Systems for Seamless Analysis and Action With OpenSe...
01h 00mPutting Apache Kafka to Use
01h 00mELK Stack - All You Need for Search In Your Applications
33mScaling your Kafka pipeline can be a pain - but it doesn’t have to be