A Year with Apache Aurora
summary
Migrates the bulk of infrastructure from Puppet-managed EC2 to Apache Aurora on Mesos, running a cluster of about 1,350 cores with over 104 worker jobs and a single Aurora config pattern plus Pants-built Python artifacts. Builds a wrapper CLI, Yama config files, template library, DeepWater ETL, Tex Runner, and service routing via Synapse/HAProxy and Zookeeper. Outcomes: drops EC2 fleet from ~773 (peaked >1,000) and cuts EC2 costs by ~33%, reduces on‑call pages, centralizes metrics/logs in OpenTSDB/Grafana and Athena.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 4:23Our approach to software engineering
- 7:08Why Mesa
- 11:49What is Aurora
- 17:47Big Decision Time
- 19:01Aurora Workflow
- 23:15Pants
- 26:04Templates
- 31:52Routing
- 35:26Consolidate
- 38:25Flume Athena
related talks
57mA Distributed and Fault-Tolerant Application with Apache Mesos & Apache Aurora
44mScaling Amazon Aurora MySQL Updates
26mHow Houghton Mifflin Harcourt Went from Months to Minutes with Infrastructure Delivery
49mHardware to AWS, EC2 to EKS: Two Years, Two Epic Migrations
01h 05mOperational excellence
21mAvoiding That $1M Dollar AWS Bill - Mark Michael, Glenn Oppegard
38mLessons Learned from the Migration to Apache Airflow
31mModernizing a megascale database