Systems @Scale Fall 2021
2021-10-13..2021-10-27
-- online --
[Systems @Scale Spring 2021, Systems @Scale Summer 2021, Systems @Scale Fall 2021, Systems @Scale Winter 2021, Systems @Scale Summer 2022, Systems @Scale Winter 2022, Systems @Scale Summer 2023, Systems @Scale 2024]
as the organisers put it
Systems @Scale is a technical conference for engineers that manage large-scale information systems serving millions of people. The operation of large-scale systems often introduces complex, unprecedented engineering challenges. The @Scale community focuses on bringing people together to discuss these challenges and collaborate on the development of new solutions. The 2021 fall series will be hosted […]
who it's for
site reliability engineers, platform engineers, data engineers, backend engineers, test engineers, engineering managers
topics
Redshift managed storage; C++ query compilation; automatic physical design; auto scaling and cross-cluster attach; Apache Flume ingestion; Beam and Dataflow DAGs; Pub/Sub and BigQuery streaming; distributed tracing resource accounting; mixed-integer resource allocation; rack-level battery alerts; cloud-native Kafka operations; integration testing with Jest
| title | author | length |
|---|---|---|
| Welcome | Lenni Kuff | 01m |
Conference opens with a host overview of the week's lineup and schedule, calling out talks from Amazon by ippo, from Twitter by lohit, and internal Facebook… | ||
| Amazon Redshift Reinvented | Ippokratis Pandis | 26m |
Launched in 2013, Redshift claims tens of thousands of customers analyzing exabytes daily and now separates compute from storage with Redshift Managed Storage… | ||
| RAS: A Resource Allowance System for Perpetual Region-wide Resource Allocation | Andy Newell | 21m |
Facebook builds a region-wide allocation system called Resource Allowance System (RAS) that replaces many server pools with a two-level allocator and a move… | ||
| Log Events @ Twitter: Challenges of Handling Billions of Events per Minute | Lohit Vijayarenu | 19m |
Pipeline scales to 5 billion events per minute and tens of petabytes per day by replacing Scribe with Apache Flume plus memory, micro‑batching and… | ||
| End-2-End Resource Accounting Leveraging Distributed Tracing | Akanksha Bansal, Richard Cornew | 17m |
A system called Transitive Resource Accounting builds on Canopy distributed traces to attach per-request CPU cost back to the originating front-end, starting… | ||
| Week 1 Live Q&A | Michael Bevilacaque-Linn | 26m |
Live Q&A covers concrete ops choices and measured outcomes: distributed tracing is valuable once services interact widely and tracing overhead is quantified… | ||
| Welcome | Lenni Kuff | 01m |
Week two announces a schedule with concrete sessions: a Lyft talk credited to ruse, a Big Spring session with Alisa and Arun, an Internet Facebook segment… | ||
| Blaming in a Blameless World | Behrooz Badii | 22m |
Lyft demonstrates that Pagerduty on-call data is far more current than a service catalog for assigning responsibility, and that standardized Envoy metrics let… | ||
| Automatic Testing for Services at Scale | Paul Marinescu | 19m |
Built an integration testing framework that instantiates production-like fragments per run, renames identifiers for namespacing identifiers, and offers… | ||
| Scaling Testing at a Startup: Integration Testing GraphQL Services with Jest | Arun, Aliisa | 20m |
BigSpring shows how Jest runs integration tests against TypeScript Node.js GraphQL resolvers using the Apollo test server (Zapier integration) and TypeORM… | ||
| Power Loss Siren: Making Facebook Resilient to Power Outages | Raghunathan Modoor Jagannathan, Sulav Malla, Parimala Kondety | 19m |
Power Loss Siren is a rack level alerting system that uses existing in rack batteries to detect outages and notify services via a relay daemon on the top of… | ||
| Week 2 Live Q&A | Jacie Fan | 37m |
Live Q&A focuses on concrete reliability practices: Facebook-grade mitigations use rack-level batteries and wait 45 seconds for generators before firing PLS… | ||
| Welcome | Lenni Kuff | 01m |
Week 3 lists the full agenda and what each slot delivers with names and order two external talks include Conflict by Yachts and databricks by Jeff followed by… | ||
| Operating Cloud-Native Kafka | Jos Visser | 25m |
Overview of running Kafka at scale in the cloud with concrete tradeoffs and engineering steps: Confluent runs thousands of managed clusters on AWS/GCP/Azure… | ||
| Making Distributed Priority Queue Disaster Ready | Jasmit Kaur Saluja, Dillon George | 23m |
FOQS uses semi-synchronous MySQL replication with primaries and two secondaries across regions plus a Shard Manager integration and a routing service that… | ||
| Lessons from Building a Large-scale, Multi-cloud Data Platform at Databricks | Jeff Pang | 19m |
Databricks runs a global multi‑cloud control plane on AWS, Azure and GCP that manages millions of VMs per day, serves 5,000+ customers including >40% of the… | ||
| Scaling Back-End Deployment | Tyler Petrochko | 19m |
Facebook's unified CD system Conveyor runs ~100,000 deployments per week with a 99.9% success rate by combining scheduled pipelines, automated Push execution… | ||
| Week 3 Live Q&A | Ayichew Hailu | 52m |
Live Q&A covers concrete system behavior and runbooks: Kafka achieves 100 MB/s on six cheap machines in tests, scales roughly linearly by adding brokers, and… | ||
| Closing Remarks | Lenni Kuff | 01m |
Recordings from the three-week series are available and posted on the at scale Facebook page, with sessions uploaded after each event and accessible for… | ||
| nothing matches | ||