基于自适应分层阈值判断的神经网络模型压缩

doi:10.19678/j.issn.1000-3428.0060042

摘要/Abstract

摘要： 面对多样化的应用环境，卷积神经网络（CNN）的架构深度不断增加以提升精度，但同时需要大量的计算参数和网络存储。针对CNN卷积层参数冗余和运算效率低的问题，提出一种基于分层阈值的自适应动态剪枝方法。设计自适应分层阈值判断算法，对批归一化层的尺度因子进行聚类分析，自适应地找到每层的分类断点并据此确定最终阈值，利用该阈值修剪正则化后的输入模型，从而避免根据经验人为定义固定阈值，减小模型尺寸和运行时占用的内存。分别采用该方法和LIU等提出的使用固定阈值且全局修剪的方法对VGGNet、ResNet、DenseNet和LeNet模型进行压缩，并在CIFAR、SVHN和MNIST数据集上测试模型性能。实验结果表明，该方法能够在模型精度与剪枝率之间找到最优平衡，剪枝后模型的测试错误率较对比方法降低0.02~1.52个百分点，同时自适应分层阈值判断算法也能避免对比方法在全局修剪时减去整个层的问题。

关键词: 深度学习, 图像识别, 卷积神经网络, 模型压缩, 网络剪枝

Abstract: Aiming at diversified application environments, the architecture depth of Convolutional Neural Network(CNN) is increasing to improve the accuracy, but at the same time, it needs a lot of computing parameters and network storage.An adaptive dynamic pruning method based on layered threshold is proposed to solve the problems of CNN convolution parameter redundancy and low operation efficiency.An adaptive hierarchical threshold judgment algorithm is designed to cluster the scale factors of the Batch Normalization(BN) layer, adaptively find the classification cluster points of each layer, and determine the final threshold accordingly.The regularized input model is trimmed by using the threshold, so as to avoid artificially defining a fixed threshold according to experience and reduce the model size and memory occupied during operation.This method and the method of using fixed threshold and global pruning proposed by LIU et al are used to compress VGGNet, ResNet, DenseNet and LeNet models respectively, and the model performance are tested on CIFAR, SVHN and MNIST data sets.Experimental results show that this method can find the optimal balance between model accuracy and pruning rate.The test error rate of the model after pruning is 0.02~1.52 percentage points lower than that of the comparison method.At the same time, the adaptive layered threshold judgment algorithm can also avoid the problem of reducing the whole layer in the global pruning of the comparison method.

Key words: deep learning, image recognition, Convolutional Neural Network(CNN), model compression, network pruning

中图分类号:

TP18

卢鹏, 万莹, 邹国良, 陈金宇, 郑宗生, 王振华. 基于自适应分层阈值判断的神经网络模型压缩[J]. 计算机工程, 2022, 48(1): 112-118,126.

LU Peng, WAN Ying, ZOU Guoliang, CHEN Jinyu, ZHENG Zongsheng, WANG Zhenhua. Neural Network Model Compression Based on Adaptive Hierarchical Threshold Judgment[J]. Computer Engineering, 2022, 48(1): 112-118,126.

https://www.ecice06.com/CN/Y2022/V48/I1/112

图/表 10

20220108122124

20220108122128

20220108122133

20220108122137

20220108122140

20220108122145

20220108122149

20220108122153

20220108122157

20220108122201

参考文献

[1] GOODFELLOW I, BENGIO Y, COURVILLE A.Deep learning[M].Cambridge, USA:MIT Press, 2016:326-366.
[2] KRIZHEVSKY A, SUTSKEVER I, HINTON G E.ImageNet classification with deep convolutional neural networks[J].Communications of the ACM, 2017, 60(6):84-90.
[3] SZEGEDY C, LIU W, JIA Y, et al.Going deeper with convolutions[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:1-9.
[4] HE K M, ZHANG X, REN S, et al.Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA:IEEE Press, 2016:1-5.
[5] ZOU Y X, YU J S, CHEN Z H, et al.Convolution neural networks model compression based on feature selection for image classification[J].Control Theory & Applications, 2017, 34(6):746-752.
[6] 叶子, 肖诗斌.卷积神经网络模型压缩在图像分类中的应用[J].北京信息科技大学学报, 2018, 33(3):52-56. YE Z, XIAO S B.Compression of convolutional neural network applied to image classification[J].Journal of Beijing Information Science & Technology University, 2018, 33(3):52-56.(in Chinese)
[7] SONG H, JEFF P, JOHN T, et al.Learning both weights and connections for efficient neural network[J].International Journal of Neural Systems, 1996, 7(2):129-147.
[8] HAN S, LIU X Y, MAO H Z, et al.EIE:efficient inference engine on compressed deep neural network[J].Computer Architecture News, 2016, 44(3):243-254.
[9] DENTON E L, ZAREMBA W, BRUNA J, et al.Exploiting linear structure within convolutional networks for efficient evaluation[C]//Proceedings of the 28th Conference on Neural Information Processing Systems.Montreal, Canada:Morgan Kaufmann Press, 2014:1269-1277.
[10] RASTEGARI M, ORDONEZ V, REDMON J, et al.XNOR-Net:ImageNet classification using binary convolutional neural networks[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2016:525-542.
[11] ZHANG X, ZOU J, MING X, et al.Efficient and accurate approximations of nonlinear convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:1984-1992.
[12] LIU Z, LI J G, SHEN Z Q.Learning efficient convolutional networks through network slimming[C]//Proceedings of the 16th IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2017:2755-2763.
[13] HUANG G, LIU Z, LAURENS V D M, et al.Densely connected convolutional networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:2261-2269.
[14] VANHOUCKE V, SENIOR A, MAO M Z.Improving the speed of neural networks on CPUs[C]//Proceedings of Deep Learning and Unsupervised Feature Learning Workshop.Cambridge, USA:MIT Press, 2011:1-4.
[15] WU J, CONG L, WANG Y, et al.Quantized convolutional neural networks for mobile devices[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:4820-4828.
[16] WEN W, WU C, WANG Y, et al.Learning structured sparsity in deep neural networks[C]//Proceedings of NIPS'16.Cambridge, USA:MIT Press, 2016:1-9.
[17] CHANGPINYO S, SANDLER M, ZHMOGINOV A.The power of sparsity in convolutional neural networks[C]//Proceedings of International Conference on Learning Representations.Toulon, France:[s.n.], 2017:1-13.
[18] HAO Z, ALVAREZ J M, PORIKLI F.Less is more:towards compact CNNs[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2016:1-16.
[19] HAN S, MAO H, DALLY W J.Deep compression:compressing deep neural networks with pruning, trained quantization and huffman coding[J].Fiber, 2015, 56(4):3-7.
[20] GUO Y, YAO A, CHEN Y.Dynamic network surgery for efficient DNNs[C]//Proceedings of NIPS'16.Cambridge, USA:MIT Press, 2016:1379-1387.
[21] SIMONYAN K, ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//Proceedings of ICLR'17.Palais, France:ICLR:2017:1-5.
[22] HE K, ZHANG X, REN S, ET AL.Deep residual learning for image recognition[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:1-12.
[23] BELLO I, ZOPH B, VASUDEVAN V, et al.Neural optimizer search with reinforcement learning[C]//Proceedings of the 34th International Conference on Machine Learning.New York, USA:ACM Press, 2017:1-16.
[24] BAKER B, GUPTA O, NAIK N.Designing neural network architectures using reinforcement learning[C]//Proceedings of ICLR'17.Palais, France:ICLR:2017:1-5.
[25] TKCVHENKO R, IZONIN I.Model and principles for the implementation of neural-like structures based on geometric data transformations[C]//Proceedings of ICCSEEA'18.Berlin, Germany:Springer, 2018:578-587.
[26] IZONIN I, TKACHENKO R, KRYVINSKA N, et al.Multiple linear regression based on coefficients identification using non-iterative SGTM neural-like structure[C]//Proceedings of the 15th International Work-Conference on Artificial Neural Networks.Gran Canaria, Spain:[s.n.], 2019:467-479.
[27] ZHU J, HASTIE T.Classification of gene microarrays by penalized logistic regression[J].Biostatistics, 2004, 5(3):427-443.
[28] MARIO Z, MICHAEL G.Accelerating K-means on the graphics processor via CUDA[C]//Proceedings of the 1st International Conference on Intensive Applications and Services.Washington D.C., USA:IEEE Press, 2009:7-15.
[29] KWAK N.Principal component analysis based on L1-norm maximization[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(9):1672-1680.
[30] SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al.Dropout:a simple way to prevent neural networks from overfitting[J].Journal of Machine Learning Research, 2014, 15(1):1929-1958.
[31] GAO H, YU S, ZHUANG L, et al.Deep networks with stochastic depth[C]//Proeedings of ECCV'16.Berlin, Germany:Springer, 2016:1-13.

选择文件类型/文献管理软件名称

选择包含的内容