Convolutional Neural Network Pruning Based on Channel Similarity Entropy

doi:10.19678/j.issn.1000-3428.0068284

Abstract

Abstract:

Convolutional Neural Network(CNN) contain a large number of filters, which occupy significant memory resources for training and storage. Pruning filters is an effective method to reduce the scale of networks, free up memory, and enhance computing speed. A primary issue with existing filter pruning methods is that they calculate the weights of filters in isolation, preserving the filter with the largest weight while often overlooking the importance of smaller weights in feature extraction. To address this, a Filter Entropy Calculation(FEC) method based on channel similarity is proposed. This method involves compressing the weight tensor by its mean value, a process whose rationality is substantiated by the structural characteristics of the filter. The similarity between channels is assessed by calculating the distance of filter channels, and filter entropy is determined based on this similarity. Filters are then ranked by their entropies, and a specific proportion of filters with lower entropy is removed. The experimental design uses different pruning ratios for different convolutional layers. Networks such as VGG-16 and ResNet-34 are pruned using the CIFAR10 and ImageNet standard datasets. Experimental results indicate that while the original accuracy is largely preserved, the number of parameters is reduced by approximately 94% and 70%, respectively. Additionally, on the Single Shot multibox Detector(SSD) framework, parameters decreased by 55.72%, and the mean Average Precision(mAP) improved by 1.04 percentage points.

Key words: Convolutional Neural Network(CNN), channel similarity, entropy, filter, pruning

摘要：

卷积神经网络(CNN)中包含大量滤波器, 参数训练以及存储占用大量内存资源。裁剪滤波器是减小网络规模、释放内存、提高计算速度的有效方法。现有滤波器裁剪方法的主要问题是将滤波器权值作为孤立的数值计算, 裁剪小权值滤波器, 保留权值大的滤波器, 忽视了部分小权值滤波器在特征提取过程中的重要性。通过分析滤波器通道之间的相似性, 提出一种基于通道相似度的滤波器熵值计算方法(FEC)。针对滤波器结构特征, 对权值张量进行均值压缩, 并证明其合理性。先计算滤波器通道距离判断通道之间的相似性, 再根据通道相似度计算滤波器熵, 由熵值大小进行滤波器排序, 删除一定比例熵值较小的滤波器。实验设计针对不同卷积层采用不同的裁剪比例, 在CIFAR10以及ImageNet标准数据集上对VGG-16和ResNet-34网络进行裁剪。实验结果表明: 在基本保持原始准确度的情况下, 分别减少了约94%和70%的参数数量; 在目标检测网络SSD上参数数量减少了55.72%, 平均精度均值(mAP)提高了1.04个百分点。

关键词: 卷积神经网络, 通道相似度, 熵, 滤波器, 裁剪

Lili GENG, Baoning NIU. Convolutional Neural Network Pruning Based on Channel Similarity Entropy[J]. Computer Engineering, 2024, 50(7): 133-143.

耿丽丽, 牛保宁. 基于通道相似度熵的卷积神经网络裁剪[J]. 计算机工程, 2024, 50(7): 133-143.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0068284

https://www.ecice06.com/EN/Y2024/V50/I7/133

Figures/Tables 17

Fig.1 Spatial position of the filter channel

Fig.2 The change of network width after pruning

Fig.3 Comparison of accuracy under different pruning ratios

Fig.4 Trend of accuracy changes

References 29

1	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60(6): 84- 90. doi: 10.1145/3065386
2	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2023-07-10]. https://arxiv.org/abs/1409.1556.
3	SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 1-9.
4	SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision[EB/OL]. [2023-07-10]. https://arxiv.org/abs/1512.00567.
5	SZEGEDY C, IOFFE S, VANHOUCKE V, et al. Inception-v4, Inception-ResNet and the impact of residual connections on learning[C]//Proceedings of the 31st AAAI Conference on Artificial Intelligence. New York, USA: ACM Press, 2017: 4278-4284.
6	HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 770-778.
7	HAN S, MAO H Z, DALLY W J. Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. Fiber, 2015, 56(4): 3- 7.
8	CARREIRA-PERPINAN M A, IDELBAYEV Y. "Learning-Compression" algorithms for neural net pruning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 8532-8541.
9	LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278- 2323. doi: 10.1109/5.726791
10	LUO J H, WU J, LIN W. ThiNet: a filter level pruning method for deep neural network compression[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 5068-5076.
11	DONG X, CHEN S Y, PAN S J. Learning to prune deep neural networks via layer-wise optimal brain surgeon[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 4860-4874.
12	MITTAL D, BHARDWAJ S, KHAPRA M M, et al. Studying the plasticity in deep convolutional neural networks using random pruning. Machine Vision and Applications, 2019, 30(2): 203- 216. doi: 10.1007/s00138-018-01001-9
13	YANG T J, CHEN Y H, SZE V. Designing energy-efficient convolutional neural networks using energy-aware pruning[C]//Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 6071-6079.
14	HU Y, SUN S, LI J, et al. A novel channel pruning method for deep neural network compression[EB/OL]. [2023-07-10]. https://arxiv.org/abs/1805.11394.
15	MOLCHANOV P, TYREE S, KARRAS T, et al. Pruning convolutional neural networks for resource efficient transfer learning[C]//Proceedings of the 5th International Conference on Learning Representations. [S. l. ]: ICLR, 2017: 1-17.
16	耿丽丽, 牛保宁. 深度神经网络模型压缩综述. 计算机科学与探索, 2020, 14(9): 1441- 1455. URL
	GENG L L, NIU B N. Survey of deep neural networks model compression. Journal of Frontiers of Computer Science and Technology, 2020, 14(9): 1441- 1455. URL
17	LI H, KADAV A, DURDANOVIC I, et al. Pruning filters for efficient ConvNets[C]//Proceedings of International Conference on Learning Representations. [S. l. ]: ICLR, 2017: 1-13.
18	LUO J H, WU J. An entropy-based pruning method for CNN compression[EB/OL]. [2023-07-10]. https://arxiv.org/abs/1706.05791v1.
19	HU H, PENG R, TAI Y W, et al. Network trimming: a data-driven neuron pruning approach towards efficient deep architectures[EB/OL]. [2023-07-10]. https://arxiv.org/abs/1607.03250v1.
20	SINGH P, VERMA V K, RAI P, et al. Leveraging filter correlations for deep model compression[C]//Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision. Washington D. C., USA: IEEE Press, 2020: 824-833.
21	ADAMCZEWSKI K, PARK M J. Dirichlet pruning for convolutional neural networks[C]//Proceedings of the 24th International Conference on Artificial Intelligence and Statistics. Washington D. C., USA: IEEE Press, 2021: 3637-3645.
22	GENG L L, NIU B N. Pruning convolutional neural networks via filter similarity analysis. Machine Learning, 2022, 111(9): 3161- 3180. doi: 10.1007/s10994-022-06193-w
23	周林勇, 谢晓尧, 刘志杰, 等. 卷积神经网络池化方法研究. 计算机工程, 2019, 45(4): 211- 216. URL
	ZHOU L Y, XIE X Y, LIU Z J, et al. Research on pooling method of convolution neural network. Computer Engineering, 2019, 45(4): 211- 216. URL
24	BUBECK S, SELLKE M. A universal law of robustness via isoperimetry. Journal of the ACM, 70(2): 10.
25	KRIZHEVSKY A. Learning multiple layers of features from tiny images[EB/OL]. [2023-07-10]. http://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz.
26	DENG J, DONG W, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2009: 248-255.
27	DONG X, CHEN S Y, PAN S J. Learning to prune deep neural networks via layer-wise optimal brain surgeon[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 4860-4874.
28	LIN S H, JI R R, YAN C Q, et al. Towards optimal structured CNN pruning via generative adversarial learning[C]//Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 2785-2094.
29	LIN M, JI R, WANG Y, et al. HRank: filter pruning using high-rank feature map[EB/OL]. [2023-07-10]. https://arxiv.org/abs/2002.10179.

[1]	ZHANG Yuxin, ZHANG Lei, OU Dongxiu. Hybrid Filtering Method for Multisource Point Cloud Data of Maglev Tracks [J]. Computer Engineering, 2024, 50(9): 54-62.
[2]	LIN Chang, GUO Wei, REN Zhecong, JIN Haibo. Unification Algorithm for Object Tracking and Segmentation Based on Transformer [J]. Computer Engineering, 2024, 50(9): 130-141.
[3]	ZHANG Lu, TIAN Chunwei, SONG Huansheng, LIU Shigang. Multi-Level Dual-Tree Complex Wavelet Network for Low-Dose CT Image Denoising [J]. Computer Engineering, 2024, 50(9): 266-275.
[4]	WANG Zhihao, QIAN Yuntao. Super-Resolution Reconstruction of Spatiotemporal Fusion for Dual-Stream Remote Sensing Images Based on Swin Transformer [J]. Computer Engineering, 2024, 50(9): 33-45.
[5]	GAO Yubao, WEN Zhicheng. Dual Decoder Image Denoising Method Based on Attention Mechanism [J]. Computer Engineering, 2024, 50(9): 324-332.
[6]	Wei GAO, Shuailong LI, Lin MAO, Lei WANG, Yingying LI, Lin HAN. One Acceleration Strategy for Operator Generation Based on TVM [J]. Computer Engineering, 2024, 50(8): 353-362.
[7]	Lei WANG, Shipeng DANG, Feng PAN. Model for Predicting Concealed Accessory Pathway Based on Convolutional Neural Network [J]. Computer Engineering, 2024, 50(8): 40-49.
[8]	Yunhe CUI, Jianpeng ZHAO, Hongzhen YANG, Xianchao LI. Curve Decision Fusion-Based Saturation Attack Detection Method in SDN [J]. Computer Engineering, 2024, 50(8): 123-132.
[9]	Jiuyuan HUO, Hongyang WANG, Tao JU, Jun HU. Research on Human Health States Assessment Model Based on AHP-EWM in Multiple Scenarios [J]. Computer Engineering, 2024, 50(7): 372-380.
[10]	Peng ZHANG, Panpan YAN, Fengjie QIAO. Attitude Assisted Correction Method for Ski Teaching Based on Long-Term Tracking [J]. Computer Engineering, 2024, 50(7): 79-86.
[11]	Ruiting NUI, Tianfeng YAN, Rui GAO, Yingzhi WANG. Deep Learning TCNN-MobileNet-Based Modulation Recognition Under Low Signal-to-Noise Radio [J]. Computer Engineering, 2024, 50(7): 204-215.
[12]	Lei ZHANG, Guochen SHEN, Dongxiu OU. Filtering Method for Feature Maps from Convolutional Neural Network for Thermal Imaging Data [J]. Computer Engineering, 2024, 50(4): 31-40.
[13]	ZHANG Lei, SHEN Guochen, OU Dongxiu. Filtering Method for Feature Maps from Convolutional Neural Network for Thermal Imaging Data [J]. Computer Engineering, 2024, 50(4): 31-40.
[14]	ZHAO Jida, ZHEN Guoyong, CHU Chengqun. Unmanned Aerial Vehicle Image Target Detection Algorithm Based on YOLOv8 [J]. Computer Engineering, 2024, 50(4): 113-120.
[15]	Bohan WANG, Xiaoyan JIANG, Liuyi FAN. Semantic Segmentation Improvement Method Based on Deep Supervision for the Construction of Latent Space [J]. Computer Engineering, 2024, 50(3): 191-199.

Please choose a citation manager

Content to export