A Verification Method on Post-Pruning Generalization Ability of Neural Network Model

doi:10.19678/j.issn.1000-3428.0052165

Computer Engineering ›› 2019, Vol. 45 ›› Issue (10): 234-238. doi: 10.19678/j.issn.1000-3428.0052165

Previous Articles Next Articles

A Verification Method on Post-Pruning Generalization Ability of Neural Network Model

LIU Chongyang, LIU Qinrang

China National Digital Switching System Engineering and Technological R & D Center, Zhengzhou 450002, China

Received:2018-07-19 Revised:2018-08-21 Online:2019-10-15 Published:2018-08-29

一种神经网络模型剪枝后泛化能力的验证方法

刘崇阳, 刘勤让

国家数字交换系统工程技术研究中心, 郑州 450002

作者简介:刘崇阳(1994-),男,硕士研究生,主研方向为人工智能、深度学习;刘勤让,研究员、博士。
基金资助:
国家科技重大专项（2016ZX01012101）；国家自然科学基金面上项目（61572520）；国家自然科学基金创新研究群体项目（61521003）。

Abstract

Abstract: To address the over-fitting problem caused by the down-regulation of the Dropout rate in the pruning operation of the neural network model,a verification method for the generalization ability of the pruning model is proposed.By artificially occluding the dataset to simulate the change of the image range,the effects of different Dropout values and pruning ratios on the accuracy of the model are analyzed,and the reasons for the change of the generalization ability of the model after pruning operation are obtained.Experiments on the convolutional neural network model lenet-5 show that the generalization ability of the pruning model is weakened because of the drop of the Dropout rate and the change of the parameter amount during the pruning operation.

Key words: Deep Neural Networks(DNN), model pruning, Deep Learning(DL), generalization ability, occlusion datasets

摘要： 针对神经网络模型在剪枝操作中Dropout率下调造成的过拟合问题，提出一种剪枝模型泛化能力的验证方法。研究人为遮挡数据集模拟图像范围的变化情况，分析不同Dropout值和剪枝比例对模型准确率的影响，进而得到剪枝操作后模型泛化能力变化的原因。在卷积神经网络模型lenet-5上进行实验，结果表明，剪枝模型泛化能力减弱是因为Dropout率下调和剪枝操作时参数量的变化。

关键词: 深度神经网络, 模型剪枝, 深度学习, 泛化能力, 遮挡数据集

CLC Number:

TP183

LIU Chongyang, LIU Qinrang. A Verification Method on Post-Pruning Generalization Ability of Neural Network Model[J]. Computer Engineering, 2019, 45(10): 234-238.

刘崇阳, 刘勤让. 一种神经网络模型剪枝后泛化能力的验证方法[J]. 计算机工程, 2019, 45(10): 234-238.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0052165

http://www.ecice06.com/EN/Y2019/V45/I10/234

Figures/Tables 14

References 15

[1]	周飞燕,金林鹏,董军.卷积神经网络研究综述[J].计算机学报,2017,40(6):1229-1251.
[2]	HAN Song.Efficient methods and hardware for deep learning[D].San Francisco,USA:Stanford University,2017.
[3]	张新钰,高洪波,赵建辉,等.基于深度学习的自动驾驶技术综述[J].清华大学学报(自然科学版),2018,58(4):438-444
[4]	HAN Song,JEFF P,JOHN T,et al.Learning both weights and connections for efficient neural networks[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2015:1135-1143.
[5]	HAN Song,MAO Huizi,DALLY W J.Deep compression:compressing deep neural networks with pruning,trained quantization and Huffman coding[EB/OL].[2018-06-18].https://arxiv.org/pdf/1510.00149.pdf.
[6]	ROSA G H D,PAPA J P,YANG X S.Handling dropout probability estimation in convolution neural networks using meta-heuristics[J].Soft Computing,2017,22(18):6147-6156.
[7]	POERNOMO A,KANG D K.Biased dropout and crossmap dropout:learning towards effective dropout regularization in convolutional neural network[J].Neural Networks,2018,104:60-67.
[8]	MESHGI K,MAEDA S I,OBA S,et al.Constructing a meta-tracker using Dropout to imitate the behavior of an arbitrary black-box tracker.[J].Neural Networks,2017,87:132-148.
[9]	SRIVASTAVA N,HINTON G,KRIZHEVSKY A,et al.Dropout:a simple way to prevent neural networks from overfitting[J].The Journal of Machine Learning Research,2014,15(1):1929-1958.
[10]	LECUN Y,BOTTOU L,BENGIO Y,et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324.
[11]	KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems.[S.l.]:Curran Associates Inc.,2012:1097-1105.
[12]	HINTON G E,SRIVASTAVA N,KRIZHEVSKY A,et al.Improving neural networks by preventing co-adaptation of feature detectors[J].Computer Science,2012,3(4):212-223.
[13]	ZHANG Chiyuan,BENGIO S,HARDT M,et al.Understanding deep learning requires rethinking generalization[EB/OL].[2018-06-18].https://arxiv.org/pdf/1611.03530.pdf.
[14]	CHEN Binghui,DENG Weihong,DU Junping.Noisy softmax:improving the generalization ability of DCNN via postponing the early softmax saturation[EB/OL].[2018-06-18].https://arxiv.org/pdf/1708.03769.pdf.
[15]	李小薪,梁荣华.有遮挡人脸识别综述:从子空间回归到深度学习[J].计算机学报,2018,41(1):177-207.

Please choose a citation manager

Content to export