Dealing with Bad Architecture Decisions: A Case Study
summary
Team builds an Elasticsearch-backed order index but discovers one minute update lag because drift takes up to a minute (sometimes 15+) to push changes into a staging table, so a database table is added as a DB search pseudo-index and queries become a two-step: search for IDs then filter in SQL. The result is a Frankenstein database+search hybrid that fixes immediacy but complicates queries, remains in production, and reveals lessons about validating backups and not locking architecture decisions.
AI-written summary · about
chapters (10)
- 0:00Start
- 1:46What orders are in progress and need to be shipped today?
- 9:00Last minute hacky solution!
- 9:26Wait, do we even need the index?
- 10:14And it's still in place one year later!
- 11:21Lesson #1: Make sure your back-up works
- 12:21Make it clear from the beginning that the Prevention architecture is open to discussion. • Hold regular retrospectives where the architecture is evaluated.
- 12:52Question 3: Why didn't I realize earlier how big of an issue this was?
- 13:47Lesson #3: Try not to panic
- 14:30Final thoughts / meta lesson
related talks
43mThe denormalized query engine design pattern
42mAvoiding Microservice Megadisaster
31m10 Costly Database Performance Mistakes (and How to Fix Them)
14mJourney of Building a Scalable, Domain-Agnostic Order Management System
49mIt’s Not Always About the Index: The Hidden Secrets of Query Performance
47mNo DB: Use a Search Engine!
36mSearch on the fly: how to lighten your Big Data
11mDatabase Evolution: Partitioning and Indexing