When Distributed Database Meets Cloud - Lessons Learned
summary
Migrates Greenplum/Apache HAWQ onto Kubernetes and builds a cloud-native HAWQ with segment pool and query CRD controllers that allocate containers per query, a hot/standby CRD for failover, and a recommender + Prometheus-based autoscaler for metrics-driven resizing. Separates storage and compute, uses PVs/S3-style object storage, and achieves multi-tenant resource sharing without per-customer DBAs. It works end-to-end but exposes hard limits: container CPU/memory sizing and per-query resource prediction remain brittle and require ongoing tuning.
AI-written summary · about
chapters (12)
- 0:00Start
- 1:23About me
- 3:32Apache HAWQ Architecture
- 13:07Summary
- 13:37Requirement
- 14:49Deployment & Operation
- 20:16Apache HAWQ on Kubernetes
- 27:12Controller
- 29:58Containerization
- 32:07Resource Management
- 32:46Resource Monitoring & Tuning
- 34:05Kubernetes Ecosystem
related talks
45mFrom Laptop to Cloud: Developing Cloud Native Applications with Containerized Databases
28mCloud Native Tech: The Building Blocks of Modern DB Software
20mThe Future of Databases
07mLightning Talk: My Database Runs on Kubernetes. What's Next? Data Platforms!
01h 12mFederated Queries with HAWQ - SQL on Hadoop and Beyond
44mRunning Non-Cloud-Native Databases in Cloud-Native Environments
44mHAWQ Architecture
26mBeyond the DBaaS Trap: Achieving Data Sovereignty With Kubernetes and CloudnativePG