Why Big Data Is Not Data Science
summary
Claim: big data is an engineering discipline separate from data science, not just “large” datasets; it centers on building and operating pipelines, storage and query layers, and production infrastructure while data science focuses on feature engineering, models and statistics. Examples: build a data lake ingesting Postgres into Kafka and HDFS and query with Impala or Presto, a credit‑scoring pipeline using pandas → TensorFlow → Flask to write scores back to PostgreSQL, and a smart‑home anomaly system using Spark streaming + Prophet. The surprising pivot: the five V's matter more than raw size and engineering work is largely invisible but crucial.
AI-written summary · about
chapters (25)
- 0:00Start
- 2:20Volume
- 3:21Velocity
- 4:28Variety
- 5:06Veracity
- 5:35Value
- 9:30Big data engineering
- 10:39Building data pipelines
- 12:50Data storage
- 13:28Data processing
- 14:03Infrastructure
- 18:54Software engineer
- 20:42Data analysis
- 21:44General ML workflow
- 23:01PRACTICAL CASES
- 23:50CASE 1
- 24:28Data lake
- 27:34CASE 2
- 28:17Credit scoring
- 31:42CASE 3
- 32:17Anomalies detection
- 36:27CASE 4
- 37:01Recommender systems
- 38:59Differences
- 40:55My contact info
related talks
44mPeople, Technology, Culture: How to make Data Science work
19mDemystifying Big Data
40mData Engineering in a Rapidly Changing World
25m"Data Engineering: The World Between Worlds"
24mGotta Feed the Beast - Data Engineering at Scale
22mDataEngBytes 2023 - BRI-T1-06 - Yanir Seroussi
08mBig Data and the Fact-Based Enterprise
36mSmall intro to Big Data