Dealing with data in dynamic systems
summary
Unit21 describes running 2 billion+ events per month and catching $100M+ in fraud while prioritizing consistency over raw availability, and lays out concrete tactics: dual writes and backfills for schema changes to guarantee transactional consistency and zero downtime, migration requirements (complete, consistent, testable, no downtime), and an ORM-based change capture wrapper over SQLAlchemy that records readable before/after user-visible events. Also covers compliance realities: 5+ year retention, auditability, background-checked personnel, and tradeoffs of overengineering analytics vs pragmatic app-level logging.
AI-written summary · about
chapters (9)
- 0:00Introduction to talk
- 1:54Fintech engineering challenges
- 4:46Legacy systems in fintech
- 6:53Compliance in fintech
- 9:20Migration strategies for data
- 15:00Balancing speed and compliance
- 18:50Data versioning pitfalls
- 21:41Application change capture
- 23:47Closing and Q&A
related talks
38mKeeping pace with data infrastructure so it doesn't outrun you
29mManaging your Data Assets: The interview
27mLive in the Data Wild West: The Data Contracts Sheriff
49mTransforming Analytical Data with Event-Driven Architecture
25mData Demons: Unleashing Chaos or Driving Performance?
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
08mReal-Time Data Systems: Speed, Scale, and Resilience
24mEvent Analytics - from unstructured to insightful