BOSH – Restore from an Availability Zone Outage

on YouTube ↗19m

" summary

Migrates director state to an external blobstore and a managed RDS database (using pg_dump), replaces static IPs with hostnames, and adds ACLs plus connection termination to isolate the old director before provisioning a new director in another AZ; blob migration needed retry logic and AWS root CA fixes. The cutover aims for zero downtime for VMs, requires rotating DB passwords and removing DNS records, and a manual restore sequence currently takes about 10–20 minutes to bring a new director up.

AI-written summary · about

" chapters (8)

1. 0:00Introduction to BOSH
2. 2:10Multi-AZ outages impact
3. 4:21Scenario of AZ outage
4. 6:49Externalizing BOSH components
5. 9:00Externalizing director database
6. 11:32Recovering from AZ outage
7. 14:25Key takeaways and future work
8. 15:49Q&A session

" related talks

• 30mStreamlining the BOSH ExperienceShatarupa Nandi, Maria Shaldibina · Cloud Foundry Summit Silicon Valley 2016
• 28mBurning Down The House: How to Plan for and Deal with Disaster Recovery in CFJatin Naik, Therese Stowell · Cloud Foundry Summit Silicon Valley 2017
• 27mBOSH: The Road AheadMorgan Fine, Danny Berger · Cloud Foundry Summit North America 2019
• 30mBeyond 1000 BOSH DeploymentsSven Schmidt · Cloud Foundry Summit Europe 2019
• 30mBurning Down the House: How to Plan for and Deal with Disaster Recovery in CFTherese Stowell · Cloud Foundry Summit Europe 2017
• 25mBOSH BootstrappingMichael Maximilien, Maria Shaldibina · Cloud Foundry Summit 2015
• 32mLossless Upgrade of BOSH DeploymentShashank Mohan Jain, Saurav Mondal · Cloud Foundry Summit North America 2019
• 35mMigrating a live site across the country without downtimeDrew Blas · MountainWest RubyConf 2013