Structural Pruning Algorithm Based on Second-Order Information of Deep Neural Network

doi:10.19678/j.issn.1000-3428.0057967

Abstract

Abstract: Most of the existing structural pruning algorithms are based on the first-order or zero-order information of Deep Neural Network(DNN).To use the second-order information of the networks for speeding up the convergence of DNN models,this paper proposes a new structural pruning algorithm based on the idea of the HAWQ algorithm.The proposed algorithm applies the power iteration method to get the max eigenvector of the Hessian matrix that is subject to the classification pre-trained network parameters.Then the obtained eigenvector is used to determine the importance of network channels and prune the channels.The pruned network parameters are slightly adjusted to improve the performance of DNN.Experimental results demonstrate that the proposed algorithm increases the classification accuracy by 0.74% when the number of network parameters is reduced by 29.9% and Floating-point Operations Per Second(FLOPS) by 34.6%,outperforming PF,LCCL and other classical pruning algorithms.

Key words: Deep Neural Network(DNN), network compression, structural pruning, second-order information, power iteration method

摘要： 现有结构化剪枝算法通常运用深度神经网络（DNN）的一阶或者零阶信息对通道进行剪枝，为利用二阶信息加快DNN网络模型收敛速度，借鉴HAWQ算法思想提出一种新的结构化剪枝算法。采用幂迭代法得到经过预训练的网络参数对应Hessian矩阵的主特征向量，据此向量衡量网络通道的重要性并进行通道剪枝，同时对剪枝后的网络参数进行微调提高DNN分类性能。实验结果表明，该算法在网络参数量和每秒浮点运算次数分别减少29.9%和34.6%的情况下，在ResNet110网络上的分类准确率提升了0.74%，剪枝效果优于PF、LCCL等经典剪枝算法。

关键词: 深度神经网络, 网络压缩, 结构化剪枝, 二阶信息, 幂迭代法

CLC Number:

TP183

JI Fanfan, YANG Xin, YUAN Xiaotong. Structural Pruning Algorithm Based on Second-Order Information of Deep Neural Network[J]. Computer Engineering, 2021, 47(2): 12-18.

季繁繁, 杨鑫, 袁晓彤. 基于深度神经网络二阶信息的结构化剪枝算法[J]. 计算机工程, 2021, 47(2): 12-18.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0057967

http://www.ecice06.com/EN/Y2021/V47/I2/12

Figures/Tables 4

References 27

[1]	KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet classification with deep convolutional neural networks[J].Communications of the ACM,2017,60(6):84-90.
[2]	LECUN Y,DENKER J S,SOLLA S A.Optimal single-class classification strategies[C]//Proceedings of the 19th International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2007:598-605.
[3]	HASSIBI B,STORK D G.Second order derivatives for network pruning:optimal brain surgeon[C]//Proceedings of the 2nd order derivatives for network pruning:optimal brain surgeon.Berlin,Germany:Springer,1993:164-171.
[4]	HAN S,POOL J,TRAN J,et al.Learning both weights and connections for efficient neural networks[EB/OL].[2020-02-13].https://arxiv.org/abs/1506.02626.
[5]	HAN S,MAO H,DALLY W J.Deep compression:compressing deep neural networks with pruning,trained quantization and huffman coding[EB/OL].[2020-02-13].https://arxiv.org/abs/1510.00149.
[6]	CHOI Y,ELKHAMY M,LEE J,et al.Towards the limit of network quantization[C]//Proceedings of International Conference on Learning Representations.Sydney,Australia:[s.n.],2017:1-8.
[7]	ZHANG Xiangyu,ZOU Jianhua,MING Xiang,et al.Efficient and accurate approximations of nonlinear convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Berlin,Germany:Springer,2015:1984-1992.
[8]	ZHANG Xiangyu,ZHOU Xinyu,LIN Mengxiao,et al.ShuffleNet:an extremely efficient convolutional neural network for mobile devices[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Berlin,Germany:Springer,2018:6848-6856.
[9]	HINTON G,VINYALS O,DEAN J.Distilling the knowledge in a neural network[EB/OL].[2020-02-13].https://arxiv.org/abs/1503.02531.
[10]	DONG Xuanyi,YANG Yi.Network pruning via transformable architecture search[C]//Proceedings of NIPS'19.Vancouver,Canada:[s.n.],2019:759-770.
[11]	LI H,KADAV A,DURDANOVIC I,et al.Pruning filters for efficient ConvNets[C]//Proceedings of International Conference on Learning Representation.Toulon,France:[s.n.],2016:1-10.
[12]	YANG Minjie,LIANG Yaling,DU Minghui.YOLO pruning algorithm based on parameter subspace and scaling factor[J/OL].Computer Engineering:1-10[2020-02-13].https://doi.org/10.19678/j.issn.1000-3428.0056932.(in Chinese)杨民杰,梁亚玲,杜明辉.基于参数子空间和缩放因子的YOLO剪枝算法[J/OL].计算机工程:1-10[2020-02-13].https://doi.org/10.19678/j.issn.1000-3428.0056932.
[13]	BOYD S,VANDENBERGHE L.Convex optimization[M].Cambridge,UK:Cambridge University Press,2004.
[14]	DONG Z,YAO Z,GHOLAMI A,et al.HAWQ:hessian aware quantization of neural networks with mixed-precision[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision.Berlin,Germany:Springer,2019:293-302.
[15]	DONG X,CHEN S,PAN S.Learning to prune deep neural networks via layer-wise optimal brain surgeon[EB/OL].[2020-02-13].https://arxiv.org/abs/1705.07565v1.
[16]	BOOTH T E.Power iteration method for the several largest eigenvalues and eigenfunctions[J].Nuclear Science and Engineering,2006,154(1):48-62.
[17]	HE Kaiming,ZHANG Xiangyu,REN Shaoqing,et al.Deep residual learning for image recognition[C]//Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition.Berlin,Germany:Springer,2016:770-778.
[18]	SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].[2020-02-13].https://arxiv.org/abs/1409.1556.
[19]	KRIZHEVSKY A,HINTON G.Learning multiple layers of features from tiny images[D].Toronto,Canada:University of Toronto,2009.
[20]	WANG Peisong.Acceleration and compression of deep neural networks[D].Beijing:University of Chinese Academy of Sciences,2018.(in Chinese)王培松.深度神经网络加速与压缩方法研究[D].北京:中国科学院大学,2018.
[21]	GUO Yiwen,YAO Anbang,CHEN Yurong.Dynamic network surgery for efficient DNNs[C]//Proceedings of Advances in Neural Information Processing Systems.Barcelona,Spain:[s.n.],2016:1379-1387.
[22]	LUO Jianhao,WU Jianxin,LIN Weiyao.ThiNet:a filter level pruning method for deep neural network compression[C]//Proceedings of 2017 IEEE International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2017:5068-5076.
[23]	DONG Xuanyi,HUANG Junshi,YANG Yi,et al.More is less:a more complicated network with less inference complexity[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:1895-1903.
[24]	YANG He,KANG Guoliang,DONG Xuanyi,et al.Soft filter pruning for accelerating deep convolutional neural networks[EB/OL].[2020-02-13].https://arxiv.org/abs/1808.06866.
[25]	LIU Zhuang,SUN Mingjie,ZHOU Tinghui,et al.Rethinking the value of network pruning[EB/OL].[2020-02-13].https://arxiv.org/abs/1810.05270.
[26]	HE Yihui,LIN Ji,LIU Zhijian,et al.AMC:AutoML for model compression and acceleration on mobile devices[C]//Proceedings of European Conference on Computer Vision.Berlin,Germany:Springer,2018:815-832.
[27]	PASZKE A,GROSS S,MASSA F,et al.PyTorch:an imperative style,high-performance deep learning library[EB/OL].[2020-02-13].https://arxiv.org/abs/1912.01703?context=cs.LG.

Please choose a citation manager

Content to export