基于通道相似度熵的卷积神经网络裁剪

doi:10.19678/j.issn.1000-3428.0068284

摘要/Abstract

摘要：

卷积神经网络(CNN)中包含大量滤波器, 参数训练以及存储占用大量内存资源。裁剪滤波器是减小网络规模、释放内存、提高计算速度的有效方法。现有滤波器裁剪方法的主要问题是将滤波器权值作为孤立的数值计算, 裁剪小权值滤波器, 保留权值大的滤波器, 忽视了部分小权值滤波器在特征提取过程中的重要性。通过分析滤波器通道之间的相似性, 提出一种基于通道相似度的滤波器熵值计算方法(FEC)。针对滤波器结构特征, 对权值张量进行均值压缩, 并证明其合理性。先计算滤波器通道距离判断通道之间的相似性, 再根据通道相似度计算滤波器熵, 由熵值大小进行滤波器排序, 删除一定比例熵值较小的滤波器。实验设计针对不同卷积层采用不同的裁剪比例, 在CIFAR10以及ImageNet标准数据集上对VGG-16和ResNet-34网络进行裁剪。实验结果表明: 在基本保持原始准确度的情况下, 分别减少了约94%和70%的参数数量; 在目标检测网络SSD上参数数量减少了55.72%, 平均精度均值(mAP)提高了1.04个百分点。

关键词: 卷积神经网络, 通道相似度, 熵, 滤波器, 裁剪

Abstract:

Convolutional Neural Network(CNN) contain a large number of filters, which occupy significant memory resources for training and storage. Pruning filters is an effective method to reduce the scale of networks, free up memory, and enhance computing speed. A primary issue with existing filter pruning methods is that they calculate the weights of filters in isolation, preserving the filter with the largest weight while often overlooking the importance of smaller weights in feature extraction. To address this, a Filter Entropy Calculation(FEC) method based on channel similarity is proposed. This method involves compressing the weight tensor by its mean value, a process whose rationality is substantiated by the structural characteristics of the filter. The similarity between channels is assessed by calculating the distance of filter channels, and filter entropy is determined based on this similarity. Filters are then ranked by their entropies, and a specific proportion of filters with lower entropy is removed. The experimental design uses different pruning ratios for different convolutional layers. Networks such as VGG-16 and ResNet-34 are pruned using the CIFAR10 and ImageNet standard datasets. Experimental results indicate that while the original accuracy is largely preserved, the number of parameters is reduced by approximately 94% and 70%, respectively. Additionally, on the Single Shot multibox Detector(SSD) framework, parameters decreased by 55.72%, and the mean Average Precision(mAP) improved by 1.04 percentage points.

Key words: Convolutional Neural Network(CNN), channel similarity, entropy, filter, pruning

耿丽丽, 牛保宁. 基于通道相似度熵的卷积神经网络裁剪[J]. 计算机工程, 2024, 50(7): 133-143.

Lili GENG, Baoning NIU. Convolutional Neural Network Pruning Based on Channel Similarity Entropy[J]. Computer Engineering, 2024, 50(7): 133-143.

https://www.ecice06.com/CN/Y2024/V50/I7/133

图/表 17

图1 滤波器通道空间位置

Fig.1 Spatial position of the filter channel

图2 裁剪后网络宽度变化

Fig.2 The change of network width after pruning

图3 不同裁剪比例下网络准确度比较

Fig.3 Comparison of accuracy under different pruning ratios

图4 准确度变化趋势

Fig.4 Trend of accuracy changes

参考文献 29

1	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60(6): 84- 90. doi: 10.1145/3065386
2	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2023-07-10]. https://arxiv.org/abs/1409.1556.
3	SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 1-9.
4	SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision[EB/OL]. [2023-07-10]. https://arxiv.org/abs/1512.00567.
5	SZEGEDY C, IOFFE S, VANHOUCKE V, et al. Inception-v4, Inception-ResNet and the impact of residual connections on learning[C]//Proceedings of the 31st AAAI Conference on Artificial Intelligence. New York, USA: ACM Press, 2017: 4278-4284.
6	HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 770-778.
7	HAN S, MAO H Z, DALLY W J. Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. Fiber, 2015, 56(4): 3- 7.
8	CARREIRA-PERPINAN M A, IDELBAYEV Y. "Learning-Compression" algorithms for neural net pruning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 8532-8541.
9	LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278- 2323. doi: 10.1109/5.726791
10	LUO J H, WU J, LIN W. ThiNet: a filter level pruning method for deep neural network compression[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 5068-5076.
11	DONG X, CHEN S Y, PAN S J. Learning to prune deep neural networks via layer-wise optimal brain surgeon[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 4860-4874.
12	MITTAL D, BHARDWAJ S, KHAPRA M M, et al. Studying the plasticity in deep convolutional neural networks using random pruning. Machine Vision and Applications, 2019, 30(2): 203- 216. doi: 10.1007/s00138-018-01001-9
13	YANG T J, CHEN Y H, SZE V. Designing energy-efficient convolutional neural networks using energy-aware pruning[C]//Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 6071-6079.
14	HU Y, SUN S, LI J, et al. A novel channel pruning method for deep neural network compression[EB/OL]. [2023-07-10]. https://arxiv.org/abs/1805.11394.
15	MOLCHANOV P, TYREE S, KARRAS T, et al. Pruning convolutional neural networks for resource efficient transfer learning[C]//Proceedings of the 5th International Conference on Learning Representations. [S. l. ]: ICLR, 2017: 1-17.
16	耿丽丽, 牛保宁. 深度神经网络模型压缩综述. 计算机科学与探索, 2020, 14(9): 1441- 1455. URL
	GENG L L, NIU B N. Survey of deep neural networks model compression. Journal of Frontiers of Computer Science and Technology, 2020, 14(9): 1441- 1455. URL
17	LI H, KADAV A, DURDANOVIC I, et al. Pruning filters for efficient ConvNets[C]//Proceedings of International Conference on Learning Representations. [S. l. ]: ICLR, 2017: 1-13.
18	LUO J H, WU J. An entropy-based pruning method for CNN compression[EB/OL]. [2023-07-10]. https://arxiv.org/abs/1706.05791v1.
19	HU H, PENG R, TAI Y W, et al. Network trimming: a data-driven neuron pruning approach towards efficient deep architectures[EB/OL]. [2023-07-10]. https://arxiv.org/abs/1607.03250v1.
20	SINGH P, VERMA V K, RAI P, et al. Leveraging filter correlations for deep model compression[C]//Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision. Washington D. C., USA: IEEE Press, 2020: 824-833.
21	ADAMCZEWSKI K, PARK M J. Dirichlet pruning for convolutional neural networks[C]//Proceedings of the 24th International Conference on Artificial Intelligence and Statistics. Washington D. C., USA: IEEE Press, 2021: 3637-3645.
22	GENG L L, NIU B N. Pruning convolutional neural networks via filter similarity analysis. Machine Learning, 2022, 111(9): 3161- 3180. doi: 10.1007/s10994-022-06193-w
23	周林勇, 谢晓尧, 刘志杰, 等. 卷积神经网络池化方法研究. 计算机工程, 2019, 45(4): 211- 216. URL
	ZHOU L Y, XIE X Y, LIU Z J, et al. Research on pooling method of convolution neural network. Computer Engineering, 2019, 45(4): 211- 216. URL
24	BUBECK S, SELLKE M. A universal law of robustness via isoperimetry. Journal of the ACM, 70(2): 10.
25	KRIZHEVSKY A. Learning multiple layers of features from tiny images[EB/OL]. [2023-07-10]. http://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz.
26	DENG J, DONG W, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2009: 248-255.
27	DONG X, CHEN S Y, PAN S J. Learning to prune deep neural networks via layer-wise optimal brain surgeon[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 4860-4874.
28	LIN S H, JI R R, YAN C Q, et al. Towards optimal structured CNN pruning via generative adversarial learning[C]//Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 2785-2094.
29	LIN M, JI R, WANG Y, et al. HRank: filter pruning using high-rank feature map[EB/OL]. [2023-07-10]. https://arxiv.org/abs/2002.10179.

[1]	王志浩, 钱沄涛. 基于Swin Transformer的双流遥感图像时空融合超分辨率重建[J]. 计算机工程, 2024, 50(9): 33-45.
[2]	李俊俊, 董建刚, 李坤. 基于Kubernetes的集群节能策略研究[J]. 计算机工程, 2024, 50(9): 82-91.
[3]	张鲁, 田春伟, 宋焕生, 刘侍刚. 用于低剂量CT图像去噪的多级双树复小波网络[J]. 计算机工程, 2024, 50(9): 266-275.
[4]	高煜宝, 文志诚. 基于注意力机制的双路解码器图像去噪方法[J]. 计算机工程, 2024, 50(9): 324-332.
[5]	王蕾, 党时鹏, 潘丰. 基于卷积神经网络的隐匿性旁路预测模型[J]. 计算机工程, 2024, 50(8): 40-49.
[6]	张朋, 严盼盼, 乔凤杰. 基于长时跟踪的滑雪教学姿态辅助矫正方法[J]. 计算机工程, 2024, 50(7): 79-86.
[7]	张洋, 刘畅, 李少青. 基于可控制性度量的图神经网络门级硬件木马检测方法[J]. 计算机工程, 2024, 50(7): 164-173.
[8]	牛瑞婷, 严天峰, 高锐, 王映植. 低信噪比下基于深度学习TCNN-MobileNet的调制识别[J]. 计算机工程, 2024, 50(7): 204-215.
[9]	张溢文, 蔡满春, 陈咏豪, 朱懿, 姚利峰. 融合空间特征的多尺度深度伪造检测方法[J]. 计算机工程, 2024, 50(7): 240-250.
[10]	逯焕宇, 张永宏, 马光义, 谢东林, 田伟. 基于半监督对抗学习的遥感图像水体提取[J]. 计算机工程, 2024, 50(7): 251-263.
[11]	火久元, 王虹阳, 巨涛, 胡军. 多场景下基于AHP-EWM的人体健康状态评估模型研究[J]. 计算机工程, 2024, 50(7): 372-380.
[12]	于洋, 孙芳芳, 吕华, 李扬, 王晓民. 基于多尺度时空注意力网络的微表情检测方法[J]. 计算机工程, 2024, 50(6): 228-235.
[13]	张雷, 沈国琛, 欧冬秀. 用于热成像数据的卷积神经网络特征图筛选方法[J]. 计算机工程, 2024, 50(4): 31-40.
[14]	张雷, 沈国琛, 欧冬秀. 用于热成像数据的卷积神经网络特征图筛选方法[J]. 计算机工程, 2024, 50(4): 31-40.
[15]	李政学, 李枝名, 彭德中, 陈杰. 基于特征对比学习和图卷积的社交网络用户分类[J]. 计算机工程, 2024, 50(4): 258-266.

选择文件类型/文献管理软件名称

选择包含的内容