Supply chain: Data infrastructure pipeline
summary
Pipeline breaks when vendor count jumps from three to 500+ vendors so the team standardizes ingestion, converts all inputs to a daily cadence, and partitions S3 by vendor/month/year. Separate per‑vendor pipelines prevent cascading failures; extraction‑stage alerts, schema validation, retries, and a dead letter Q capture bad events. Reconciliation of row counts and checksums plus anomaly detection restore trust. The result is resilient, scalable supply‑chain ingestion that keeps ML features and revenue metrics accurate.
AI-written summary · about
chapters (8)
- 0:00Data infrastructure in supply chains
- 1:14Data quality and infrastructure
- 3:00Real-time challenge at Amazon
- 4:25Scaling pipeline for vendors
- 6:31Vendor injection case takeaways
- 7:57Pipeline design for growth
- 9:03Monitoring and data quality
- 10:30Data-driven supply chain
related talks
32mReliability in Finance: Delivering Quality Market Data at Scale
49mScaling Software Supply Chain Source Security in Large Enterprises
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
32m7 steps to build-your-own data pipeline - for day 1 of your startup
35mEffective Handling of Third-Party Supplier Incidents
10mBuilding High-Performance Financial Data Pipelines
41m7 steps to build your own data pipeline
29mData Acquisition: Overcoming Challenges