Simplified distributed training with tf.distribute parameter servers
summary
Explains how ParameterServerStrategy gives asynchronous, data-parallel training for large-embedding or preemptible-machine workloads, using a single-client setup where a coordinator uses ClusterCoordinator to create per-worker datasets, call coordinator.schedule for steps, and coordinator.join to sync completion. Shows minimal code changes from MirroredStrategy or Keras model.fit, describes variables living on parameter servers, workers pulling latest weights and sending gradients, and that the approach scales and tolerates worker preemption.
AI-written summary · about
chapters (4)
related talks
38mParameter Server on Flink, an approach for model-parallel machine learning
35mLarge Scale Distributed Deep Learning on Kubernetes Clusters
08mPyTorch Distributed: Towards Large Scale Training
35mLarge Scale Distributed Deep Learning on Kubernetes Clusters
27mTips and tricks for distributed large model training
29mLarge Scale Distributed Deep Learning with Kubernetes Operators
24mFast training and near-linear scaling with DataParallel in Amazon SageMaker
31mFederated Machine Learning with Kubernetes