Apache Spark for Big Data Processing

on YouTube ↗01h 25m

" summary

Overview of an actual pipeline using Apache Spark 1.3–1.5 with Java 8 and Twitter4J: live tweets stream in via a reliable Spring XD receiver, get batched into 2 second batches (block interval ~200ms), processed with Spark Streaming + Spark SQL, features built with character bigrams and HashingTF, and a saved MLlib k-means model clusters three-language k-means (English, Spanish, Japanese). Demos include local wordcount and HDFS writes, Spark runs on Standalone/YARN, integration handles lifecycle and works with some demo hiccups but produces usable results.

AI-written summary · about

" chapters (37)

1. 0:00Intro
2. 0:21Agenda
3. 1:39Speakers
4. 2:52Machine Learning
5. 7:19Resilient Distributed Datasets (RDDs)
6. 8:24Access data
7. 9:04Deploy
8. 9:46Global view
9. 11:11An example: the word count
10. 11:32Entry point: the Spark context
11. 12:54Count the words
12. 13:54Filter on words
13. 16:14How to detect the language of a tweet?
14. 17:05Twitter streaming connection
15. 17:27Starting point: Spark streaming context
16. 18:31Creating a DStream
17. 19:15Save the data
18. 23:07Spark SQL & DataFrame
19. 24:07SQL context
20. 24:36Creating a DataFrame
21. 25:20DataFrames operations
22. 26:20Running SQL queries
23. 27:41Main algorithms provided MLlib Classification & Regression
24. 28:06Other.. Feature extraction and transformation
25. 28:49Clustering: K-means algorithm
26. 29:14Load the data using DataFrames
27. 29:52Select only 3 languages
28. 30:10Working on the tweet's content
29. 30:49Features extraction and transformation
30. 31:37Extract the features: N-gram & HashingTF
31. 35:45Other interesting projects
32. 36:59Focus on Apache Spark's 1.5 release
33. 37:51Why should I care about Spark?
34. 48:26Spark streaming failure scenarios
35. 52:00Spark integration in Spring XD
36. 55:01Advantages when using with Spring XD
37. 59:29Advantages...

" related talks

• 44mData Science at Scale with Apache SparkDean Wampler · GOTO Chicago 2015
• 52m100% Big Data. 0% Hadoop. 0% Java.Pavlo Baron · Tech Mesh 2012
• 35mPySpark : Combining Machine Learning & Big DataAyon Roy · Data Con LA 2020
• 37mBig Data Analytics, Machine Learning, Real Time ProcessingKai Waehner · Codemotion Milan 2016
• 01h 47mRadanalytics.io streaming applications workshopMichael McCune, William Benton · DevConf.CZ 2019
• 01h 48mBig Data made easy with a Spark (3.0)Jean-Georges Perrin · All Things Open 2020
• 30mDeploy end to end ML pipeline using Apache spark ...Nick Pentreath, Prashant Sharma · Scale By The Bay 2019
• 11mScaling R with SparkJavier Luraschi · rstudio::conf 2019