~/talk

Of Mice & Elephants

on YouTube 30m
Talkat SREcon21 · 2021-10-12..10-14· [data-engineering]

speakers

name:Koon Seng Lim
links:none on file
name:Sandeep Hooda
links:none on file

resources

video:https://youtu.be/s0C3bOFXI6E
slides:none on file

summary

A production job writes thousands-deep nested paths and triggers constant failures that, combined with an object-store firmware bug, causes the cluster to be offline for nearly five days while teams copy millions of tiny files out one-by-one. Investigation finds 88% files <5MB, builds a SonarQube plugin to catch bad queries, adds a spark-sampling utility to compute optimal repartitioning, re-architects with object-backed tiers and experiments with Ozone, runs 281 chaos tests, and reports ~96% provisioning and ~60% batch runtime gains.

AI-written summary · about

chapters (10)

  1. 0:00Introduction
  2. 1:29Data Lake
  3. 7:55Small Files
  4. 14:05SRE retrospective
  5. 14:28Data ingestion
  6. 16:05SRE principles
  7. 18:00Govermation
  8. 20:01Leap Engineering
  9. 24:30Collaboration
  10. 25:28Learning Development

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.