A Fast and Progressive Convolutional Neural Architecture Search Algorithm

doi:10.19678/j.issn.1000-3428.0063354

Abstract

Abstract: Manually designing a Convolutional Neural Network(CNN) architecture is extremely difficult and requires a high level of professionalism.The gradient differentiable search is fast and efficient.However, this method has some drawbacks, such as a large gap and low stability between search and evaluation.To address these issues, a Fast and Progressive Neural Architecture Search(FPNAS) algorithm that combines progressive search and greedy index is proposed. Using the greedy index as the edge selection criterion effectively closes the deep gap and improves the relevance and stability of the search evaluation by gradually expanding the architecture of the search stage.Simultaneously, the method of dividing datasets is used to reduce search time to address problems of high computing resource consumption of neural architecture search algorithms.The experimental results show that the stability of the CNN architecture searched by FPNAS is improved when compared to the PDARTS and SGAS algorithms, using accuracy and search time as indicators.The search time is reduced by 0.19 and 0.14 GPU Days, respectively.On the CIFAR-10 dataset, the maximum accuracy reaches 97.7%.

Key words: deep learning, Convolutional Neural Network(CNN), differentiable architecture search, progressive architecture search, dateset dividing method

摘要： 手动设计卷积神经网络结构对专业性要求高、难度大。基于梯度可微的搜索快速高效，但这类方法存在深度鸿沟和稳定性较差的问题。提出一种结合渐进式搜索和贪心指标的快速渐进式结构搜索算法（FPNAS），通过渐进式扩展搜索阶段的结构，使得搜索阶段的网络结构逐渐接近评估阶段，避免深度鸿沟造成的影响。同时，通过运用贪心指标作为选边准则，增加搜索评估的相关性并提高搜索的稳定性。针对网络结构搜索算法消耗计算资源多的问题，提出渐进式划分数据集方法，通过分阶段不同比例的数据集划分来减少结构搜索的计算资源消耗。以准确率和搜索时间作为评价指标，将FPNAS与渐进式可微结构搜索算法和贪心搜索算法进行对比，实验结果表明，FPNAS搜索出的网络结构稳定性得到改进，搜索时间分别缩短0.19和0.14个GPU Days，在CIFAR-10数据集上精度最高达到97.7%。

关键词: 深度学习, 卷积神经网络, 可微结构搜索, 渐进式结构搜索, 划分数据集方法

CLC Number:

TP183

ZHAO Liang, FANG Wei. A Fast and Progressive Convolutional Neural Architecture Search Algorithm[J]. Computer Engineering, 2022, 48(12): 134-139,149.

赵亮, 方伟. 一种快速渐进式卷积神经网络结构搜索算法[J]. 计算机工程, 2022, 48(12): 134-139,149.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0063354

http://www.ecice06.com/EN/Y2022/V48/I12/134

Figures/Tables 5

References

[1] LECUN Y, BENGIO Y, HINTON G.Deep learning[J].Nature, 2015, 521(7553):436-444.
[2] LECUN Y, BOTTOU L, BENGIO Y, et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE, 1998, 86(11):2278-2324.
[3] KRIZHEVSKY A, SUTSKEVER I, HINTON G E.ImageNet classification with deep convolutional neural networks[J].Communications of the ACM, 2017, 60(6):84-90.
[4] SIMONYAN K, ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].[2021-09-21].https://arxiv.org/abs/1409.1556.
[5] SZEGEDY C, LIU W, JIA Y Q, et al.Going deeper with convolutions[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:1-9.
[6] HE K M, ZHANG X Y, REN S Q, et al.Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:770-778.
[7] HUANG G, LIU Z, VAN DER MAATEN L, et al.Densely connected convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:2261-2269.
[8] HOWARD A G, ZHU M L, CHEN B, et al.MobileNets:efficient convolutional neural networks for mobile vision applications[EB/OL].[2021-09-21].https://arxiv.org/abs/1704.04861.
[9] HU J, SHEN L, SUN G.Squeeze-and-excitation networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:7132-7141.
[10] WATKINS C J C H.Learning from delayed rewards[D].London, UK:King's College, 1989.
[11] ZOPH B, LE Q V.Neural architecture search with reinforcement learning[EB/OL].[2021-09-21].https://arxiv.org/abs/1611.01578.
[12] ZOPH B, VASUDEVAN V, SHLENS J, et al.Learning transferable architectures for scalable image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:8697-8710.
[13] 王军, 冯孙铖, 程勇.深度学习的轻量化神经网络结构研究综述[J].计算机工程, 2021, 47(8):1-13. WANG J, FENG S C, CHENG Y.Survey of research on lightweight neural network structures for deep learning[J].Computer Engineering, 2021, 47(8):1-13.(in Chinese)
[14] BAKER B, GUPTA O, NAIK N, et al.Designing neural network architectures using reinforcement learning[EB/OL].[2021-09-21].https://arxiv.org/abs/1611.02167.
[15] ZHONG Z, YAN J J, WU W, et al.Practical block-wise neural network architecture generation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:2423-2432.
[16] REAL E, AGGARWAL A, HUANG Y P, et al.Regularized evolution for image classifier architecture search[C]//Proceedings of AAAI Conference on Artificial Intelligence.Palo Alto, USA:AAAI Press, 2019:4780-4789.
[17] SUN Y N, XUE B, ZHANG M J, et al.Completely automated CNN architecture design based on blocks[J].IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(4):1242-1254.
[18] HONG Z, FANG W, SUN J, et al.A fast GA for automatically evolving CNN architectures[C]//Proceedings of Genetic and Evolutionary Computation Conference.Washington D.C., USA:IEEE Press, 2020:213-214.
[19] LU Z C, WHALEN I, BODDETI V, et al.NSGA-Net:neural architecture search using multi-objective genetic algorithm[C]//Proceedings of Genetic and Evolutionary Computation Conference.Washington D.C., USA:IEEE Press, 2019:419-427.
[20] XIE L X, YUILLE A.Genetic CNN[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2017:1388-1397.
[21] SUN Y N, XUE B, ZHANG M J, et al.Automatically designing CNN architectures using the genetic algorithm for image classification[J].IEEE Transactions on Cybernetics, 2020, 50(9):3840-3854.
[22] LIU H X, SIMONYAN K, YANG Y M.DARTS:differentiable architecture search[EB/OL].[2021-09-21].https://arxiv.org/abs/1806.09055.
[23] CHEN X, XIE L X, WU J, et al.Progressive differentiable architecture search:bridging the depth gap between search and evaluation[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2019:1294-1303.
[24] LI G H, QIAN G C, DELGADILLO I C, et al.SGAS:sequential greedy architecture search[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:1617-1627.
[25] AHMED K, TORRESANI L.Connectivity learning in multi-branch networks[EB/OL].[2021-09-21].https://arxiv.org/abs/1709.09582.
[26] SHIN R, PACKER C, SONG D.Differentiable neural network architecture search[C]//Proceedings of International Conference on Learning Representations.Vancouver, Canada:[s.n.], 2018:1-10.
[27] 杨晓梅, 郭文强, 张菊玲.考虑多样性的深度神经网络结构搜索方法研究[J].计算机工程, 2020, 46(12):105-112, 133. YANG X M, GUO W Q, ZHANG J L.Research on structure search method for deep neural network considering diversity[J].Computer Engineering, 2020, 46(12):105-112, 133.(in Chinese)
[28] DEVRIES T, TAYLOR G W.Improved regularization of convolutional neural networks with cutout[EB/OL].[2021-09-21].https://arxiv.org/abs/1708.04552.
[29] CAI H, CHEN T Y, ZHANG W N, et al.Efficient architecture search by network transformation[C]//Proceedings of AAAI Conference on Artificial Intelligence.Palo Alto, USA:AAAI Press, 2018:1-10.
[30] LIU H X, SIMONYAN K, VINYALS O, et al.Hierarchical representations for efficient architecture search[EB/OL].[2021-09-21].https://arxiv.org/abs/1711.00436.
[31] REAL E, MOORE S, SELLE A, et al.Large-scale evolution of image classifiers[C]//Proceedings of International Conference on Machine Learning.[S.l.]:PMLR, 2017:2902-2911.

Please choose a citation manager

Content to export