Top 10 data engineering mistakes #bbuzz
summary
Pick a proper workflow orchestrator rather than ad hoc scripts—use Luigi or Airflow—so pipelines are reproducible and recoverable (typical rollback time with batch + orchestrator is ~30 minutes). Don’t use Hadoop/stream tech as a one-size-fits-all online DB; dumping production DBs or reading years of raw data daily causes huge load (one Cassandra extraction grew into a 25 hours spike that knocked out logins). Plan for petabytes when erasure is required, enforce schemas to avoid 25+ timestamp formats, prefer incremental algorithms, and separate offline experiments from online serving.
AI-written summary · about
chapters (12)
- 0:00Introduction and purpose
- 2:54Batch processing pitfalls
- 6:27Misusing Hadoop for real-time
- 11:32Functional principles in architecture
- 15:49Separating online and offline systems
- 18:22Avoiding imaginary scaling
- 21:14Stream processing trade-offs
- 25:30Monitoring data quality
- 28:30Pipeline change inertia
- 30:44Privacy and governance costs
- 34:00Focus on business value
- 35:27Q&A
related talks
34mAvoiding Big Data Antipatterns
45mData pipelines from zero to solid
30mData Engineering building your BI infrastructure from scratch
25mData Demons: Unleashing Chaos or Driving Performance?
45mValidating Big Data Jobs - Stopping Failures before Production
57mBACK TO THE FUTURE: EMERGING TRENDS IN DATA ENGINEERING
47mAll the DataOps, all the paradigms
45mResilient Predictive Data Pipelines