Democratising data analytics
summary
An on‑premises analytics platform is built on Kubernetes, Spark and Airflow with an S3 data lake (Parquet now, migrating to Iceberg by year end), a semantic SQL layer using Dremio as single entry, and DataHub for catalog and lineage. It writes about 25 TB/day into ~120 TB of storage, scans 1.1 million rows/second on average, enforces governance via infrastructure‑as‑code and git workflows, and reports successful organization‑wide democratization and reproducible production pipelines.
AI-written summary · about
chapters (10)
- 0:00Start
- 0:15Christian Thiel
- 2:20Digital Transformation
- 2:49End-to-End Analytics
- 3:56Open Data Platform
- 4:47Scalability and Automation
- 5:51Overview of the Platform
- 8:45Data Catalog
- 13:59Business Change Management
- 15:12Examples of the Use Cases of Businesses or Companies or Governments That Use the Data
related talks
26mRe-architecting the Data Platform with Spark on Kubernetes
34mThe Data Analytics Platform or How to Make Data Science in a Box Possible
35mBuilding a data analytics platform on strong opinions held loosely
41mOn Data: Sky above, sand below, peace within
30mData Engineering building your BI infrastructure from scratch
37mBuild a Modern Data Platform on AWS
34mTales of Data Architecture Evolution
50mDesigning a scalable data platform based on Apache Spark/Iceberg