Jorge A. Castañón, Data Scientist, IBM
summary
RedRock is an iPad app built in 10 days that queries a 20‑node SoftLayer Hadoop/Spark cluster via REST and returns results in seconds. It trains word2vec + k‑means on about six terabytes (≈350M tweets, 30 days) to produce a 90k × 100‑dim vectors matrix, trained in ~3 hours on a cluster; min_count≈5 is used. Spark MLlib handles word vectors and clustering, producing labeled clusters (NFL, hockey, LA Times) and cosine‑similarity lookups that power the visualizations and search.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:38Outline
- 1:16Experiment
- 2:05Data
- 2:42Design Thinking
- 3:11Understand
- 4:36Prototype
- 5:17Evaluate
- 5:51App Architecture
- 7:30Word to Beck
- 9:58Measure Similarity
- 10:41Experimentation
- 14:12Pipeline
- 15:00Red Rock
- 15:17Twitter
- 16:54Volume of Tweets
- 17:08Users by Profession
- 17:21Sentiment Analysis
- 18:48Conclusion
related talks
44mData Science at Scale with Apache Spark
51mSpotlight on Emerging Data Science Worskspaces and Spark as Development Levers
01h 48mBig Data made easy with a Spark (3.0)
34mThe Data Analytics Platform or How to Make Data Science in a Box Possible
01h 26mData Driven Action: a primer on Data Science
01h 25mApache Spark for Big Data Processing
24mJoseph K. Bradley, Software Engineer, Databricks Inc. @ MLconf SEA
26mOptimal Approaches for Real-Time Machine Learning with Apache Spark on Kubernetes.