~/talk

Streamlining Big Data Workflows with Memoization & Work Avoidance

on YouTube 22m
Talkat ArgoCon NA 2023 · on Monday, 2023-11-06· [machine-learning, frontend-architecture]

speakers

name:Alan Clucas
links:none on file
name:Julie Vogelman
links:none on file

resources

video:https://youtu.be/29r3wZHP2eU
slides:none on file

summary

Argo Workflows uses memoization via ConfigMaps to skip steps entirely by storing a key, outputs and a max_age so repeated runs drop pod startup time to zero on cache hits; artifacts store paths while parameters embed values and risk the ~1MB etcd configmap limit and eviction. As an alternative, work avoidance with marker files keeps logic in the container and still requires pod startup. Pure, input‑derived steps are safe to cache; nonpure steps need careful keys and expiration.

AI-written summary · about

chapters (13)

  1. 0:00Introduction
  2. 1:06Memoization reduces work
  3. 2:31Memoization in Argo Workflows
  4. 4:15Memoization keys skip steps
  5. 6:23Memoizing workflows and DAGs
  6. 7:27ConfigMaps caching limits
  7. 9:16Memoized DNA sequencing example
  8. 11:27Work avoidance technique
  9. 13:13ConfigMaps size limits
  10. 15:03Pure steps memoization rules
  11. 16:54Risks of impure steps
  12. 17:16Further reading
  13. 17:37Q&A

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.