Modern Data Stack: Optimising and Scaling Data in a Tech Company
summary
Argues that adopting DataOps and a dedicated Data+Analytics Engineering function saves engineering time and frees analysts for higher‑value work and shows a practical Sololearn stack: FiveTran + Databricks for ingestion, BigQuery as serverless warehouse, dbt tests and GitHub pull‑request gating. Describes bronze/silver/gold layers, Confluent Cloud for streaming, Power BI for reporting, batch + half‑hour orchestration, and claims a working data platform built in under six months supporting 300+ data models with automated quality checks.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:12Company vision and data
- 2:38What is DataOps
- 5:50DataOps challenges
- 8:17Solutions and infrastructure
- 12:13Modern Data Stack benefits
- 16:46SolarLearn’s Data Stack
- 21:04Pipeline layers bronze silver gold
- 26:37Future enhancements
- 28:24Data team roles
- 32:18DataOps with product teams
- 34:32Q&A
related talks
51mWorkflows of Highly Functional App & Data Engineering Teams
41mModern Data Architectures – What They Are and How to Implement?
36mThe MODERN Modern Data Stack: Building an Open Distributed Data Warehouse Beyond...
43mScaling Data Stacks: How To Build a Great Data Model
58mUnderstanding Big Data for Software Engineers
25mPlatform Engineering for Modern Data Infrastructure
28mPractical steps to establish and scale your data & ML team
47mHow to pick the best Data transformation strategy