LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Scheduling

Nanda K. Unnikrishnan, Keshab K. Parhi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The time required for training the neural networks increases with size, complexity, and depth. Training model parameters by backpropagation inherently creates feedback loops. These loops hinder efficient pipelining and scheduling of the tasks within the layer and between consecutive layers. Prior approaches, such as PipeDream, have exploited the use of delayed gradient to achieve inter-layer pipelining. However, these approaches treat the entire backpropagation as a single task; this leads to an increase in computation time and processor underutilization. This paper presents novel optimization approaches where the gradient computations with respect to the weights and the activation functions are considered independently; therefore, these can be computed in parallel. This is referred to as intra-layer optimization. Additionally, the gradient computation with respect to the activation function is further divided into two parts and distributed to two consecutive layers. This leads to balanced scheduling where the computation time of each layer is the same. This is referred to as inter-layer optimization. The proposed system, referred to as LayerPipe, reduces the number of clock cycles required for training while maximizing processor utilization with minimal inter-processor communication overhead. LayerPipe achieves an average speedup of 25% and upwards of 80% with 7 to 9 processors with less communication overhead when compared to PipeDream.

Original languageEnglish (US)
Title of host publication2021 40th IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2021 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665445078
DOIs
StatePublished - 2021
Event40th IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2021 - Munich, Germany
Duration: Nov 1 2021Nov 4 2021

Publication series

NameIEEE/ACM International Conference on Computer-Aided Design, Digest of Technical Papers, ICCAD
Volume2021-November
ISSN (Print)1092-3152

Conference

Conference40th IEEE/ACM International Conference on Computer-Aided Design, ICCAD 2021
Country/TerritoryGermany
CityMunich
Period11/1/2111/4/21

Bibliographical note

Funding Information:
This research was supported in part by the National Science Foundation under grant number CCF-1954749.

Publisher Copyright:
© 2021 IEEE.

Fingerprint

Dive into the research topics of 'LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Scheduling'. Together they form a unique fingerprint.

Cite this