Week 2 Q&A
summary
Live Q&A covers operational specifics: SLO tooling (Slick) supports configurable alerting, auto-generated availability/latency SLOs, MySQL+Hive storage, batch pipelines with two weeks full-granularity plus real-time detection windows. Megatron is a supercomputer-optimized transformer stack running a 530B parameters model, forcing data curation and moving from fp16 to bfloat16 precision and reducing costly activation checkpointing via memory-efficiency work. Remediation pipeline uses supervised ML rollouts, a six-month rolling window of repair data, blocklists for low-confidence actions, multi-step remediation flows, and technician escalation when failure causes are unclear.
AI-written summary · about
chapters (17)
- 0:00Welcome
- 1:27SLO adoption and alerting
- 2:31Megatron vs DeepSpeed
- 3:36Balancing alarms and safety
- 4:41Out-of-the-box SLIs and SLOs
- 5:22Handling unknown failures
- 6:49Training 530B model challenges
- 8:17ML model rollout in hardware
- 9:01Slick team roles
- 9:43Megatron improvements
- 11:09Hardware vs software failures
- 12:55Hive to MySQL switch
- 13:38Multi-step remediations
- 14:40Repair model training data
- 16:23Low-confidence suggestions
- 17:07Slick data pipeline frequency
- 18:10Closing remarks
related talks
32mSystems @Scale Summer Live Q&A Session #4
01h 00mSystems @Scale Summer Q&A Week 4
26mDataEngBytes 2023 - MEL-T2-05 - Paul Brebner
24mUnder Pressure: Applying ML in Real Time
20mOpAMP in Action: User Configurable Observability Pipelines
35mSystems Scale Summer Q&A Week 1
01h 16mInterview and Q&A
01mintro video