Process Migration in the Orchestration World

on YouTube ↗39m

" summary

Demonstrates live container checkpoint and restart on an Apache Mesos mini‑cluster using runc and DMTCP, checkpointing a Redis instance and preserving an in‑memory key across failure and restore; checkpoint file is ~2.9 MB in the demo and periodic checkpointing is set to 5s. Notes that migration cost scales with memory (1 GB → ~10 s at 100 MB/s) and that Mesos currently lacks integrated task states for checkpoint/restore so integration work remains.

AI-written summary · about

" chapters (14)

1. 0:00Start
2. 0:38Overview
3. 1:12Stateless vs. Stateful Applications
4. 2:28Scheduling Stateless vs. Statelul Applications
5. 4:27How to Improve Scheduling of Stateful Applications?
6. 5:28Terminology
7. 6:16How to Migrate a Process/Container/VM?
8. 8:41What is Checkpointing?
9. 11:30Checkpointing Use Cases
10. 14:07Stateful Applications with Checkpointing
11. 17:03The datacenter abstraction
12. 17:48Resource offers
13. 22:16Mesos mechanics are fair
14. 34:18Future Work: Checkpointing as a Service

" related talks

• 52mNightmares of a Container Orchestration SystemJörg Schad · DevOpsCon 2017 Berlin
• 35mFault Tolerant Frameworks - Making Use of CNI Without DockerAaron Wood · MesosCon North America 2017
• 15mKeynote: State of the MesosBen Hindman · MesosCon North America 2017
• 31mScaling a Distributed Container Management System: Diego PerformanceJames Myers, Luan Dos Santos · Cloud Foundry Summit Silicon Valley 2016
• 42mStreaming Data Pipelines on Apache Mesos: Lessons LearnedDean Wampler · MesosCon North America 2017
• 17mMeces: Latency-efficient Rescaling via Prioritized State Migration for Stateful...Rong Gu, Han Yin, Weichang Zhong, Chunfeng Yuan, Yihua Huang · USENIX ATC '22
• 18mIn-Memory Key-Value Store Live Migration with NetMigrateZaoxing Liu, Zeying Zhu, Yibo Zhao · 22nd USENIX Conference on File and Storage Technologies (FAST '24)
• 41mEfficient cluster resource management by using Cook and MesosLi Jin · HighLoad++ 2015