Data Engineering building your BI infrastructure from scratch
summary
Built a daily batch BI pipeline that pulls Segment events from a Redshift-based data lake into a new Redshift data warehouse modeled as a star schema, transforms with Apache Spark on EMR and schedules jobs with Airflow on Kubernetes, and exposes reports in Tableau. It delivers daily analytics but initially misses events, erodes trust, and costs one engineer six months before hiring help. Production lessons: enforce data quality (accuracy, completeness, timeliness) with tools like Great Expectations, consider CDC/streaming or a lakehouse earlier.
AI-written summary · about
chapters (12)
- 0:00Introduction and agenda
- 1:03Speaker introduction and background
- 2:05Data disciplines and roles explained
- 5:56Building BI infrastructure at Hotjar
- 8:02Initial infrastructure challenges
- 12:12Data warehouse and reporting setup
- 12:53Importance of data quality
- 16:06Lessons learned from the project
- 19:56Team and time management insights
- 21:41Recommendations for streaming adoption
- 22:02Advice for different teams
- 25:28Q&A and closing remarks
related talks
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications
58mTales of Data Architecture Evolution
43mData warehousing reinvented for today's needs
34mTales of Data Architecture Evolution
17mDemocratising data analytics
40mData Engineering in a Rapidly Changing World
45mBuilding an Analytics Platform
42mWeb Development To Big Data. A Journey