Systems @Scale Spring 2021
as the organisers put it
Systems @Scale is a technical conference for engineers that manage large-scale information systems serving millions of people. The operation of large-scale systems often introduces complex, unprecedented engineering challenges. The @Scale community focuses on bringing people together to discuss these challenges and collaborate on the development of new solutions. The 2021 series will be hosted virtually. […]
who it's for
site reliability engineers, backend engineers, platform engineers, storage engineers, distributed systems engineers, data engineers, engineering managers
topics
distributed priority queue with sharded MySQL; sendfile-backed TCP replication; semantic video replication; mega volumes and semantic volumes; cloud filesystem on S3; LSM metadata for snapshots; content-addressable deduplicated clumps; workflow engines with transfer queues; DAC-style dependency workflows; ticket-based read-your-writes; virtual consensus and log-structured protocols; mergeable summaries for analytics
| title | author | length |
|---|---|---|
| Scaling a Distributed Priority | Niharika Devanathan, Akshay Nanavati | 18m |
A horizontally scalable distributed priority queue that uses sharded MySQL with transactional writes and secondary indexes, a Fox router that keeps in‑memory… | ||
| Breaking Magical Barriers | Gerhard Lazu | 26m |
Claim: a single replicated RabbitMQ queue reaches 1,000,000 messages/sec (up from ~25k) by changing the replication and I/O plumbing rather than the VM or… | ||
| Optimizing Video Storage Via Semantic Replication | Chidambaram Muthu, Dan Danaila | 18m |
Facebook treats different encodings of the same video as semantic replicas and compacts cold videos by keeping one physical copy per encoding instead of two… | ||
| Multi-Purpose Cloud Filesystem | Sazzala Reddy | 22m |
SaaS DR replaces a second data center by streaming incremental backups into a cloud filesystem that launches VMware clusters on demand; steady state stores… | ||
| Workflows@Facebook: Powering Developer Productivity and Automation at Facebook Scale | Girish Joshi, Dan Shiovitz | 27m |
Facebook runs two complementary engines: a dependency layer called DAC that sits on Fox distributed priority queue with MySQL state and shard manager, and an… | ||
| Designing a Workflow Engine From First Principles | Maxim Fateev | 23m |
Temporal is a workflow engine built to preserve full program state durably and run millions of bounded workflows by scaling out shards rather than scaling up… | ||
| FlightTracker: Social Graph Consistency at Scale | Xiao Shi, Scott Pruett | 24m |
FlightTracker enforces user-centric read‑your‑writes by issuing compact tickets that list recent object keys and versions and attaching them to reads so stores… | ||
| Virtualizing Consensus in Delos for Rapid Upgrades and Happy Engineers | Mahesh Balakrishnan | 19m |
Delos claims online, zero-downtime upgrades by Virtual Consensus: a virtual log abstraction that lets a ZooKeeper-backed shim be swapped for a native consensus… | ||
| Mergeable Summaries for High Scale, High Cardinality Analytics | Peter Ballis | 25m |
Describes a practical system that turns commutative associative merge operators into mergeable summaries used at Microsoft, Sisu, and in Apache Druid to answer… | ||
| Order And Consistency At Scale | Daniel Chia | 22m |
Spanner synchronously replicates ranges with Paxos and enforces external consistency so concurrent transactions appear serializable and ordered by real time… | ||
| nothing matches | ||