Approximate Random Dropout for DNN training acceleration in GPGPU
Zhuoran Song1, Ru Wang1, Dongyu Ru1, Zhenghao Peng1, Hongru Huang1, Hai Zhao1, Xiaoyao Liang1 and Li Jiang1,2
1Shanghai Jiao Tong University
2MoE Key Lab of Artificial Intelligence, AI Institute
ABSTRACT
The training phases of Deep neural network (DNN) consumes enormous processing time and energy. Compression techniques utilizing the sparsity of DNNs can effectively accelerate the inference phase of DNNs. However, it can be hardly used in the training phase because the training phase involves dense matrix-multiplication using General Purpose Computation on Graphics Processors (GPGPU), which endorse regular and structural data layout. In this paper, we propose the Approximate Random Dropout that replaces the conventional random dropout of neurons and synapses with a regular and online generated patterns to eliminate the unnecessary computation and data access. We develop a SGD-based Search Algorithm that producing the distribution of dropout patterns to compensate the potential accuracy loss. We prove our approach is statistically equivalent to the previous dropout method. Experiments results on multilayer perceptron (MLP) and long short-term memory (LSTM) using well-known benchmarks show that the speedup rate brought by the proposed Approximate Random Dropout ranges from 1.18- 2.16 (1.24-1.85) when dropout rate is 0.3-0.7 on MLP (LSTM) with negligible accuracy drop.
Keywords: Training, Neural network, Dropout.
