作者投稿和查稿 主编审稿 专家审稿 编委审稿 远程编辑

计算机工程 ›› 2021, Vol. 47 ›› Issue (2): 12-18. doi: 10.19678/j.issn.1000-3428.0057967

• 热点与综述 • 上一篇    下一篇

基于深度神经网络二阶信息的结构化剪枝算法

季繁繁, 杨鑫, 袁晓彤   

  1. 南京信息工程大学 江苏省大数据分析技术重点实验室, 南京 210044
  • 收稿日期:2020-04-03 修回日期:2020-05-13 出版日期:2021-02-15 发布日期:2020-05-07
  • 作者简介:季繁繁(1995-),男,硕士研究生,主研方向为机器学习、神经网络;杨鑫,硕士研究生;袁晓彤,教授、博士。
  • 基金资助:
    国家自然科学基金(61876090,61936005);国家新一代人工智能重大项目(2018AAA0100400)。

Structural Pruning Algorithm Based on Second-Order Information of Deep Neural Network

JI Fanfan, YANG Xin, YUAN Xiaotong   

  1. Jiangsu Key Laboratory of Big Data Analysis Technology, Nanjing University of Information Science and Technology, Nanjing 210044, China
  • Received:2020-04-03 Revised:2020-05-13 Online:2021-02-15 Published:2020-05-07

摘要: 现有结构化剪枝算法通常运用深度神经网络(DNN)的一阶或者零阶信息对通道进行剪枝,为利用二阶信息加快DNN网络模型收敛速度,借鉴HAWQ算法思想提出一种新的结构化剪枝算法。采用幂迭代法得到经过预训练的网络参数对应Hessian矩阵的主特征向量,据此向量衡量网络通道的重要性并进行通道剪枝,同时对剪枝后的网络参数进行微调提高DNN分类性能。实验结果表明,该算法在网络参数量和每秒浮点运算次数分别减少29.9%和34.6%的情况下,在ResNet110网络上的分类准确率提升了0.74%,剪枝效果优于PF、LCCL等经典剪枝算法。

关键词: 深度神经网络, 网络压缩, 结构化剪枝, 二阶信息, 幂迭代法

Abstract: Most of the existing structural pruning algorithms are based on the first-order or zero-order information of Deep Neural Network(DNN).To use the second-order information of the networks for speeding up the convergence of DNN models,this paper proposes a new structural pruning algorithm based on the idea of the HAWQ algorithm.The proposed algorithm applies the power iteration method to get the max eigenvector of the Hessian matrix that is subject to the classification pre-trained network parameters.Then the obtained eigenvector is used to determine the importance of network channels and prune the channels.The pruned network parameters are slightly adjusted to improve the performance of DNN.Experimental results demonstrate that the proposed algorithm increases the classification accuracy by 0.74% when the number of network parameters is reduced by 29.9% and Floating-point Operations Per Second(FLOPS) by 34.6%,outperforming PF,LCCL and other classical pruning algorithms.

Key words: Deep Neural Network(DNN), network compression, structural pruning, second-order information, power iteration method

中图分类号: