Hybrid Parallelism

DAPPLE: A Pipelined Data Parallel Approach for Training Large Models
We propose DAPPLE, a synchronous training framework which combines data parallelism and pipeline parallelism for large DNN models. It features a novel parallelization strategy planner to solve the partition and placement problems, and explores the optimal hybrid strategy of data and pipeline parallelism. We also propose a new runtime scheduling algorithm…
Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, LanSong Diao, Xiaoyong Liu, Wei Lin
PDF Code Source Document
DAPPLE: A Pipelined Data Parallel Approach for Training Large Models