Data-driven production apps

on YouTube ↗39m

" summary

Shopify uses its Rails app data to build Capital, a data-driven lending product that has issued about $100 million to ~5,000 merchants. Production tables are ETL’d into a separate warehouse, cleaned and organized with dimensional modeling (facts and denormalized dimensions / star schema) so daily sales and risk metrics are precomputed. Experiments and ML run in the warehouse, insights are pushed back into the app as snapshot records, and schema guards plus monitoring and a Slack bot enforce resiliency and human verification before automation.

AI-written summary · about

" chapters (11)

1. 0:00Shopify Capital intro
2. 3:53Legacy apps need upgrades
3. 11:01Adding ML to Rails apps
4. 16:00ETL to warehouse
5. 19:36Dimensional modeling
6. 26:40Long-term with dimensional data
7. 28:45Experiments in warehouse
8. 31:37Insights to production
9. 33:25Resilient data pipelines
10. 37:24Automation vs verification
11. 38:05Q&A

" related talks

• 41mDeconstructing the MonolithKirsten Westeinde · Craft Conference 2022
• 53mBusiness transformation through data and MLStepan Pushkarev, Russell Lamb · Amazon re:MARS 2022
• 21mData-driven approaches in a technology startupMichal Szczecinski · Rakuten Technology Conference 2018
• 26mBuilding A Domain-driven Data Strategy at Checkout.comAlexandre Carvalho · Big Data LDN 2022
• 20mMigrating a Large Scale Search Dataset in Production in a Highly Available...Leila Vayghan · SREcon24 Americas
• 13mData for productAbdelilah Moulida · BlaBlaConf 2022
• 17mSpeeding Data Product Delivery from Model to MarketplaceMike O Donnell · SQLBits 2024
• 28mUsing Dataops To Build And Manage Data Products On SnowflakeOmar Khawaja, Guy Adams, Paul Rankin · Big Data LDN 2022