面向轻量级卷积网络的激活函数与压缩模型

doi:10.19678/j.issn.1000-3428.0061550

计算机工程 ›› 2022, Vol. 48 ›› Issue (5): 242-250. doi: 10.19678/j.issn.1000-3428.0061550

面向轻量级卷积网络的激活函数与压缩模型

徐增敏^1,3, 陈凯², 郭威伟^1,4, 赵汝文¹, 蒋占四⁵

1. 桂林电子科技大学数学与计算科学学院, 广西桂林 541004;
2. 杭州海康威视数字技术股份有限公司, 杭州 310052;
3. 桂林安维科技有限公司, 广西桂林 541010;
4. 中国通信建设集团设计院有限公司第四分公司, 郑州 450052;
5. 桂林电子科技大学机电工程学院, 广西桂林 541004

收稿日期:2021-05-06 修回日期:2021-07-04 发布日期:2022-05-10
作者简介:徐增敏(1981—),男,副教授、博士,主研方向为人工智能、计算机视觉、人体行为分析;陈凯(通信作者),工程师;郭威伟,学士;赵汝文,讲师、硕士;蒋占四,教授、博士生导师。
基金资助:
国家自然科学基金“视频侦查中基于深度学习的人体行为识别技术研究”（61862015）；广西重点研发计划项目“面向涉密场所的视频人体行为分析系统研发及应用”（AB17195025）；广西高校中青年教师科研基础能力提升项目“基于手机指纹识别身份认证系统研究”（2019KY0253）。

Activation Function and Compression Model for Lightweight Convolutional Network

XU Zengmin^1,3, CHEN Kai², GUO Weiwei^1,4, ZHAO Ruwen¹, JIANG Zhansi⁵

1. School of Mathematics and Computing Science, Guilin University of Electronic and Technology, Guilin, Guangxi 541004, China;
2. Hangzhou Hikvision Digital Technology Co., Ltd, Hangzhou, 310052, China;
3. Guilin Anview Technology Co., Ltd., Guilin, Guangxi 541010, China;
4. The fourth branch of China Communications Construction Group Design Institute Co., Ltd, Zhengzhou, 450052, China;
5. School of Mechanical and Electrical Engineering, Guilin University of Electronic and Technology, Guilin, Guangxi 541004, China

Received:2021-05-06 Revised:2021-07-04 Published:2022-05-10

摘要/Abstract

摘要： 因卷积神经网络参数膨胀，导致模型训练时占用大量的计算资源和存储资源，从而限制其在边缘终端上的应用。依据深度可分离卷积模型MobileNet V1的设计思路，结合自门控函数和ReLU函数的特点，构建一种改进的激活函数和压缩神经网络模型MobileNet-rhs。将ReLU函数和swish函数分别作为分段线性函数，设计激活函数ReLU-h-swish，通过优化卷积单元结构，解决模型训练过程中难以激活部分神经元的问题，以减少特征信息丢失。构建一种剔除卷积核的压缩模型，从模型深处自下而上剔除2ⁿ个卷积核，减少逐点卷积的参数量。在CIFAR-10和CIFAR-100数据集上进行实验，结果表明，引入ReLU-h-swish函数构建MobileNet-rhs模型的Top-1分类准确率为80.38%。相比MobileNet-rhs模型，压缩后MobileNet-rhs模型的参数量减少17.9%，其Top-1分类准确率仅降低2.28个百分点。此外，利用Tensorflow将该模型部署在安卓平台上，实现图像分类相册的应用。

关键词: manifold of interest变换, 深度可分离卷积, 逐点卷积, 自门控函数, Kotlin协程

Abstract: The abundance of computing and storage resources required in model training to relieve the parameter expansion of a deep convolution neural network, limiting a network's application on edge terminals.Based on the design idea of the depthwise separable convolution model MobileNet V1, this study proposes an improved activation function and compressed neural network model that combines the characteristics of the self-gating function and the ReLU function.Taking the ReLU function and swish function as piecewise linear functions, the activation function ReLU-h-swish is designed.By optimizing the convolution unit structure to reduce the loss of feature information, it is difficult to activate some neurons during the process of model training.A compression model with convolution kernels removed is constructed.To compress the model, 2ⁿ convolution kernels are removed from the depths of the model from bottom to top to reduce the number of parameters of point-to-point convolution.The experimental results on the CIFAR-10 and CIFAR-100 datasets show that the Top-1 classification accuracy of the MobileNet-rhs model constructed by introducing the ReLU-h-swish function is 80.38%.Compared to the MobileNet-rhs model, the parameters of the compressed MobileNet-rhs model are reduced by 17.9%, and the Top-1 classification accuracy is reduced by only 2.28 percentage points.In addition, Tensorflow is used to deploy the model on the Android platform, which realizes the application of image classification album.

Key words: transformation of manifold of interest, depthwise seperable convolution, pointwise convolution, self-gating function, Kotlin coroutine

中图分类号:

TP391.41

徐增敏, 陈凯, 郭威伟, 赵汝文, 蒋占四. 面向轻量级卷积网络的激活函数与压缩模型[J]. 计算机工程, 2022, 48(5): 242-250.

XU Zengmin, CHEN Kai, GUO Weiwei, ZHAO Ruwen, JIANG Zhansi. Activation Function and Compression Model for Lightweight Convolutional Network[J]. Computer Engineering, 2022, 48(5): 242-250.

https://www.ecice06.com/CN/Y2022/V48/I5/242

图/表 17

20220806174608

20220806174612

20220806174615

20220806174619

20220806174622

20220806174625

20220806174629

20220806174633

20220806174637

20220806174642

20220806174645

20220806174650

20220806174654

20220806174658

20220806174702

20220806174706

20220806174712

参考文献

[1] KRIZHEVSKY A, SUTSKEVER I, HINTON G E.ImageNet classification with deep convolutional neural networks[J].Communications of the ACM, 2017, 60(6):84-90.
[2] SIMONYAN K, ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].[2021-04-03].https://arxiv.org/pdf/1409.1556.pdf.
[3] SZEGEDY C, LIU W, JIA Y Q, et al.Going deeper with convolutions[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:1-9.
[4] HE K M, ZHANG X Y, REN S Q, et al.Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:770-778.
[5] IANDOLA F N, HAN S, MOSKEWICZ M W, et al.SqueezeNet:AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size[EB/OL].[2021-04-03].https://arxiv.org/pdf/1602.07360.pdf.
[6] HOWARD A G, ZHU M L, CHEN B, et al.MobileNets:efficient convolutional neural networks for mobile vision applications[EB/OL].[2021-04-03].https://arxiv.org/pdf/1704.04861.pdf.
[7] ZHANG X Y, ZHOU X Y, LIN M X, et al.ShuffleNet:an extremely efficient convolutional neural network for mobile devices[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:6848-6856.
[8] FERRARI V, HEBERT M, SMINCHISESCU C, et al.Unsupervised class-specific deblurring[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Spring, 2018:258-374.
[9] 崔婷婷, 唐林波, 衡勇, 等.基于轻量化卷积神经网络的空中红外目标识别[C]//第十二届全国信号和智能信息处理与应用学术会议论文集.北京, 中国:中国学术期刊(光盘版)电子杂志社, 2018:5. CUI T T, TANG L B, HENG Y, et al.Airborne infrared target recognition based on lightweight convolutional neural network[C]//Proceedings of the 12th National Conference on Signal and Intelligent Information Processing and Application.Beijing, China:China Academic Journal (CD version) Electronic Magazine, 2018:5.(in Chinese)
[10] 曹昭睿, 白帆, 刘凤丽, 等.基于轻量化神经网络的目标识别跟踪算法研究[J].弹箭与制导学报, 2020, 40(1):19-23. CAO Z R, BAI F, LIU F L, et al.Design of target recognizing and tracking algorithm based on tiny convolution neural network[J].Journal of Projectiles, Rockets, Missiles and Guidance, 2020, 40(1):19-23.(in Chinese)
[11] 刘俊, 姜涛, 徐小康, 等.基于轻量化深度网络的舰船目标识别技术研究[J].无线电工程, 2019, 49(12):1025-1030. LIU J, JIANG T, XU X K, et al.Research on ship target recognition technology based on lightweight deep network[J].Radio Engineering, 2019, 49(12):1025-1030.(in Chinese)
[12] 付佐毅, 周世杰, 李顶根.轻量级目标识别深度神经网络及其应用[J].计算机工程与应用, 2020, 56(18):131-136. FU Z Y, ZHOU S J, LI D G.Lightweight target recognition deep neural network and its application[J].Computer Engineering and Applications, 2020, 56(18):131-136.(in Chinese)
[13] 李亚辉.面向舰船目标识别应用的关键技术研究[D].杭州:杭州电子科技大学, 2019. LI Y H.Research on key technologies for ship target recognition application[D].Hangzhou:Hangzhou Dianzi University, 2019.(in Chinese)
[14] SANDLER M, HOWARD A, ZHU M L, et al.MobileNetV2:inverted residuals and linear bottlenecks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:4510-4520.
[15] 徐梦婕.轻量化地标感知算法及认知地图中的应用[D].成都:电子科技大学, 2020. XU M J.Lightweight landmark perception algorithm and its application in cognitive MAP[D].Chengdu:University of Electronic Science and Technology of China, 2020.(in Chinese)
[16] YANG Y T, YANG R Z, PAN L H, et al.A lightweight deep learning algorithm for inspection of laser welding defects on safety vent of power battery[J].Computers in Industry, 2020, 123:103306.
[17] QASAIMEH M, AL-QASSAS R S, MOHAMMAD F, et al.A novel simplified AES algorithm for lightweight real-time applications:testing and discussion[J].Recent Advances in Computer Science and Communications, 2020, 13(3):435-445.
[18] RAJAKUMAR M P, RAMYA J, MAHESWARI B U.Health monitoring and fault prediction using a lightweight deep convolutional neural network optimized by Levy flight optimization algorithm[J].Neural Computing and Applications, 2021, 33(19):12513-12534.
[19] HE K M, ZHANG X Y, REN S Q, et al.Delving deep into rectifiers:surpassing human-level performance on ImageNet classification[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2015:1026-1034.
[20] LI C L, RAVANBAKHSH S, POCZOS B.Annealing Gaussian into ReLU:a new sampling strategy for leaky-ReLU RBM[EB/OL].[2021-04-03].https://arxiv.org/pdf/1611.03879.pdf.
[21] RAMACHANDRAN P, ZOPH B, LE Q V.Searching for activation functions[EB/OL].[2021-04-03].https://arxiv.org/pdf/1710.05941.pdf.
[22] HOWARD A, SANDLER M, CHEN B, et al.Searching for MobileNetV3[C]//Proceedings of IEEE/CVF International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2019:1314-1324.
[23] GOODFELLOW I, BENGIO Y, COURVILLE A.Deep learning[M].Cambridge, USA:MIT Press, 2016.
[24] GLOROT X, BORDES A, BENGIO Y.Deep sparse rectifier neural networks[C]//Proceedings of the 14th International Conference on Artificial Intelligence and Statistics.[S.l]:AAAI Press, 2011:315-323.
[25] ABADI M, AGARWAL A, BARHAM P, et al.TensorFlow:large-scale machine learning on heterogeneous distributed systems[EB/OL].[2021-04-03].https://arxiv.org/abs/1603.04467.

选择文件类型/文献管理软件名称

选择包含的内容

面向轻量级卷积网络的激活函数与压缩模型

Activation Function and Compression Model for Lightweight Convolutional Network

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	王娅茹, 唐璐, 陈爱斌, 彭伟雄, 沈平. 基于LPDMR-NET的鸟鸣声识别[J]. 计算机工程, 2024, 50(10): 174-184.
[2]	曹广硕, 黄瑞章, 陈艳平, 秦永彬. 基于多模态学习的乳腺癌生存预测研究[J]. 计算机工程, 2024, 50(1): 296-305.
[3]	李松江, 耿兰兰, 王鹏. 基于改进Yolov4的车辆目标检测[J]. 计算机工程, 2023, 49(4): 272-280.
[4]	郭克友, 王苏东, 李雪, 张沫. 基于Dim env-YOLO算法的昏暗场景车辆多目标检测[J]. 计算机工程, 2023, 49(3): 312-320.
[5]	王程, 刘元盛, 刘圣杰. 基于改进YOLOv4的小目标行人检测算法[J]. 计算机工程, 2023, 49(2): 296-302,313.
[6]	史宝岱, 张秦, 李瑶, 李宇环. 面向图像目标识别的轻量化卷积神经网络[J]. 计算机工程, 2022, 48(6): 257-262.
[7]	柳聪, 屈丹, 司念文, 魏紫薇. 基于深度可分离卷积的轻量级图像超分辨率重建[J]. 计算机工程, 2022, 48(6): 228-234.
[8]	史钰祜, 张起贵. 基于局部注意的快速视频目标检测方法[J]. 计算机工程, 2022, 48(5): 314-320.
[9]	王军, 冯孙铖, 程勇. 深度学习的轻量化神经网络结构研究综述[J]. 计算机工程, 2021, 47(8): 1-13.
[10]	林杰, 陈春梅, 刘桂华, 祝礼佳. 室内服务机器人的实时场景分割算法[J]. 计算机工程, 2021, 47(7): 21-29.
[11]	胡江颢, 王红雨, 乔文超, 马靖煊. 基于轻量级网络的实时人体关键点检测算法[J]. 计算机工程, 2021, 47(4): 218-225.
[12]	江枭宇, 李忠兵, 张军豪, 彭娇, 文婷. 基于NCS2神经计算棒的车辆检测方法[J]. 计算机工程, 2021, 47(3): 298-303.
[13]	黄靖淞, 左颢睿, 张建林. 轻量化目标检测算法研究及应用[J]. 计算机工程, 2021, 47(10): 236-241.
[14]	康智慧, 王全玉, 王战军. 一种端到端的人脸对齐方法[J]. 计算机工程, 2021, 47(10): 207-213.
[15]	曹渝昆, 桂丽嫒. 基于深度可分离卷积的轻量级时间卷积网络设计[J]. 计算机工程, 2020, 46(9): 95-100,109.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

面向轻量级卷积网络的激活函数与压缩模型

Activation Function and Compression Model for Lightweight Convolutional Network

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献

相关文章 15

编辑推荐

Metrics

本文评价