~/talk

Adopting Ray @ LinkedIn

on YouTube 29m
Talkat Ray Summit 2020 · on Thursday, 2020-10-01· at 11:45· Virtual 2, Virtual· [distributed-systems, machine-learning]

speakers

name:Jonathan Hung
links:none on file
name:Nitin Pasumarthy
links:none on file

resources

video:https://youtu.be/0Z0Th9ySIfs
slides:none on file

summary

Integrates Ray on YARN via TonY and Azkaban by having the TonY application master request containers, reserve ports, broadcast a head+worker topology, and launch Ray commands (demo runs a 1-head + 3-worker cluster and executes 100 remote calls aggregated across 10 iterations). Runs on LinkedIn’s Hadoop estate (7,000-node cluster, 1.8PB compute memory, 300PB storage) and a 2,000-GPU ML cluster. Uses Ray Tune with Hyperopt, Async HyperBand, and PBT for HPO, finds PBT yields best accuracy, runs 150 experiments in 14 hours vs 15 in 6 hours, and achieves 2x faster training by reducing GPU idle time with measurable user engagement and slight revenue gains.

AI-written summary · about

chapters (3)

  1. 0:00LinkedIn ML infrastructure
  2. 4:10Running Ray clusters on Yarn
  3. 12:31Hyperparameter tuning with Ray

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.