Pipeline Parallelism

The last decade has witnessed growth in the computational requirements for training deep neural networks. Current approaches (e.g., …

We propose DAPPLE, a synchronous training framework which combines data parallelism and pipeline parallelism for large DNN models. It features a novel parallelization strategy planner to solve the partition and placement problems, and explores the optimal hybrid strategy of data and pipeline parallelism. We also propose a new runtime scheduling algorithm…