7 Tips to Get Started with Spark

on YouTube ↗44m

" summary

Practical advice for getting started with Apache Spark: prototype interactively with spark-shell, remember lazy evaluation (transformations run only when an action is called), and test locally using a SparkSession wrapper. The workflow shown moves from RDD to DataFrame to Dataset for type safety and performance, uses HDFS/YARN and the Spark History Server for debugging, minimizes and keeps UDFs pure, and highlights production issues like retries, executor failures, and duplicated work when multiple actions run.

AI-written summary · about

" chapters (4)

1. 0:00Start
2. 0:137 TIPS TO GET STARTED WITH SPARK
3. 13:18LEARN THE BASICS OF SCALA
4. 34:18KEEP AN OPEN MIND TO TEST WITH SPARK

" related talks

• 01h 00mGetting Started with SparkNikhil Nanivadekar · Devnexus 2020
• 44mData Science at Scale with Apache SparkDean Wampler · GOTO Chicago 2015
• 01h 48mBig Data made easy with a Spark (3.0)Jean-Georges Perrin · All Things Open 2020
• 59mSpark 2.0Matei Zaharia · Scala Days Berlin 2016
• 11mHow Companies are Using Spark, and Where the Edge in Big Data Will BeMatei Zaharia · Strata Conference 2014
• 51mSpotlight on Emerging Data Science Worskspaces and Spark as Development LeversCarlo Appugliese · ApacheCon North America 2017
• 01h 25mApache Spark for Big Data ProcessingLudwine Probst, Ilayaperumal Gopinathan · SpringOne2GX 2015
• 16mPetabytes of data: How to use R at scaleJeff Fletcher · satRday Johannesburg 2019