神经网络滤波器竞争训练

doi:10.19678/j.issn.1000-3428.0064345

摘要/Abstract

摘要： 非重要权重元素的修剪和重新激活可避免神经网络过度参数化，然而权重元素的重新激活一般是通过激活整个滤波器实现，分类准确率不高。针对该问题，在神经网络训练过程中提出一种滤波器权值竞争训练算法。在局部和全局范围内选择并定位劣质滤波器，根据前向匹配策略寻找相应的优质滤波器，使用其中的最优和次优权重元素交叉更新劣质滤波器中的次劣和最劣权重元素，在神经网络结构上使陷入局部极值的权值进行重新激活。实验结果表明，应用滤波器权值竞争训练算法的ResNet、DenseNet等普通神经网络在CIFAR数据集上的分类准确率和在ImageNet数据集上的Top-1准确率平均提升了0.79和1.13个百分点，MobileNet、ShuffleNet等轻量级神经网络平均提升了2.22和2.93个百分点，优于现有的滤波器竞争训练算法。

关键词: 神经网络, 权值竞争, 重新激活, 滤波器剪枝, 插件式训练

Abstract: Pruning and reactivation of unimportant weight elements can prevent overparameterization of neural networks.However, the reactivation of weight elements is typically achieved by activating the entire filter, where the classification accuracy is not high.To solve this problem, a filter weight competition training algorithm is proposed to train neural networks.In the local and global range, the low-quality filter is selected and located, and the corresponding high-quality filter based on the forward matching strategy is found.Optimal and suboptimal weight elements are used to cross-reference and update the subworst and worst weight elements in the low-quality filter.In the neural network, the weights trapped in local extrema are reactivated.Experimental results show that the classification accuracy of ordinary neural networks such as ResNet and DenseNet on the CIFAR dataset and the Top-1 accuracy on the ImageNet dataset trained by filter weight competition algorithm increase by 0.79 and 1.13 percentage points on average, respectively, as compared with the benchmark network.Similarly, lightweight neural networks such as MobileNet and ShuffleNet increase by 2.22 and 2.93 percentage points.The neural network image classification effect following filter weight competition training is thus better than the existing filter competition training algorithm.

Key words: neural network, weight competition, reactivation, filter pruning, plug-in training

中图分类号:

TP183

安志国, 彭政, 易满成, 刘健欣, 俞思帆. 神经网络滤波器竞争训练[J]. 计算机工程, 2023, 49(4): 120-124.

AN Zhiguo, PENG Zheng, YI Mancheng, LIU Jianxin, YU Sifan. Filter Competition Training of Neural Networks[J]. Computer Engineering, 2023, 49(4): 120-124.

https://www.ecice06.com/CN/Y2023/V49/I4/120

图/表 7

20230417184800

20230417184803

20230417184807

20230417184810

20230417184813

20230417184816

20230417184819

参考文献

[1] 李伟, 黄鹤鸣, 张会云, 等.基于深度多特征融合的CNNs图像分类算法[J].计算机仿真, 2022, 39(2):322-326. LI W, HUANG H M, ZHANG H Y, et al.A CNNs image classification algorithm based on deep fusion of multi-features[J].Computer Simulation, 2022, 39(2):322-326.(in Chinese)
[2] 徐海燕.基于通道相似度注意力的图像分类研究[J].信息技术与信息化, 2021(11):78-80. XU H Y.Research on image classification based on channel similarity attention[J].Information Technology and Informatization, 2021(11):78-80.(in Chinese)
[3] DENIL M, SHAKIBI B, DINH L, et al.Predicting parameters in deep learning[EB/OL].[2022-02-18].https://arxiv.org/abs/1306.0543.
[4] HAN S, MAO H, DALLY W J.Deep compression:compressing deep neural networks with pruning, trained quantization and huffman coding[EB/OL].[2022-02-18].https://arxiv.org/abs/1510.00149v4.
[5] LI H, SAMET H, KADAV A, et al.Pruning filters for efficient ConvNets[EB/OL].[2022-02-18].https://arxiv.org/abs/1608.08710v2.
[6] FRANKLE J, CARBIN M.The lottery ticket hypothesis:finding sparse, trainable neural networks[EB/OL].[2022-02-18].https://arxiv.org/abs/1803.03635.
[7] KWON S J, LEE D, KIM B, et al.Structured compression by weight encryption for unstructured pruning and quantization[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:1906-1915.
[8] LEE N, AJANTHAN T, TORR P H S.SNIP:single-shot network pruning based on connection sensitivity[EB/OL].[2022-02-18].https://arxiv.org/abs/1810.02340.
[9] HE Y, DING Y H, LIU P, et al.Learning filter pruning criteria for deep convolutional neural networks acceleration[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:2006-2015.
[10] LIN M B, JI R R, WANG Y, et al.HRank:filter pruning using high-rank feature map[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:1526-1535.
[11] LIN M B, JI R R, ZHANG Y X, et al.Channel pruning via automatic structure search[C]//Proceedings of the 29th International Joint Conference on Artificial Intelligence.New York, USA:ACM Press, 2021:673-679.
[12] 卢海伟, 袁晓彤.基于层融合特征系数的动态网络结构化剪枝[J].模式识别与人工智能, 2019, 32(11):1051-1059. LU H W, YUAN X T.Dynamic network structured pruning via feature coefficients of layer fusion[J].Pattern Recognition and Artificial Intelligence, 2019, 32(11):1051-1059.(in Chinese)
[13] 王彩玲, 王炯, 蒋国平.基于双DDPG的全局自适应滤波器剪枝方法[J].南京邮电大学学报(自然科学版), 2021, 41(5):59-66. WANG C L, WANG J, JIANG G P.Global adaptive filter pruning based on dual DDPG[J].Journal of Nanjing University of Posts and Telecommunications(Natural Science Edition), 2021, 41(5):59-66.(in Chinese)
[14] PRAKASH A, STORER J, FLORENCIO D, et al.RePr:improved training of convolutional filters[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2019:10666-10675.
[15] MENG F X, CHENG H, LI K, et al.Filter grafting for deep neural networks[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:6599-6607.
[16] JEONG J, SHIN J.Training CNNs with selective allocation of channels[EB/OL].[2022-02-18].https://arxiv.org/abs/1905.04509.
[17] HE Y, DONG X Y, KANG G L, et al.Asymptotic soft filter pruning for deep convolutional neural networks[J].IEEE Transactions on Cybernetics, 2020, 50(8):3594-3604.
[18] 李杉, 许新征.基于双角度并行剪枝的VGG16优化方法[J].计算机科学, 2021, 48(6):227-233. LI S, XU X Z.Parallel pruning from two aspects for VGG16 optimization[J].Computer Science, 2021, 48(6):227-233.(in Chinese)
[19] YE J, LU X, LIN Z, et al.Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers[EB/OL].[2022-02-18].https://arxiv.org/abs/1802. 00124.
[20] KRIZHEVSKY A.Learning multiple layers of features from tiny images[D].Toronto, Canada:University of Toronto, 2009.
[21] RUSSAKOVSKY O, DENG J, SU H, et al.ImageNet large scale visual recognition challenge[J].International Journal of Computer Vision, 2015, 115(3):211-252.
[22] HE K M, ZHANG X Y, REN S Q, et al.Identity mappings in deep residual networks[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2016.
[23] HUANG G, LIU Z, VAN DER MAATEN L, et al.Densely connected convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:2261-2269.
[24] HOWARD A G, ZHU M, CHEN B, et al.MobileNet:efficient convolutional neural networks for mobile vision applications[EB/OL].[2022-02-18].https://arxiv.org/abs/1704.04861.
[25] ZHANG X, ZHOU X, LIN M, et al.ShuffleNet:an extremely efficient convolutional neural network for mobile devices[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:6848-6856.

选择文件类型/文献管理软件名称

选择包含的内容