Customizing Parallel Formulations of Backpropagation Learning Algorithm to Neural Network Architectures: A Summary of Results

Several generic parallel formulations of the Backpropagation learning algorithm have been proposed recently. Further speedups are possible by customizing parallel formulations to the architecture of the neural network. This paper addresses the issue of customizing parallel formulations of Backpropagation learning algorithm to a given neural network architecture on multiprocessors with hypercube-like communication topology. We introduce a new parallel formulation called Rectangular Checkerboarding which adapts to the network architecture and can provide performance gains for non-uniform neural networks, where the number of nodes vary across the layers. Algebraic analysis shows that each instance of rectangular checkerboarding (using a specific rectangular processor grid) is optimal for an important family of network architectures. Experiments on CM-5 show that customizing to network architecture can provide significant (- 50%) performance gains for many interesting non-uniform neural network architectures, which are currently used in important applications. We also introduce the Staircase framework, which can use different processor grids for different layers of a neural network.