Review of Research on Neural Architecture Search Algorithms Based on Non-Gradient Evolution

doi:10.19678/j.issn.1000-3428.0057520

Abstract

Abstract: Automated deep learning is one of the new research hotspots in the field of deep learning.Neural architecture search algorithms are frequently used for the implementation of automated deep learning,as they can automatically design neural network structure by defining different search space,search strategy or optimization strategy.This paper introduces the development history of evolutionary algorithms and evolutionary neural networks.Then it introduces different methods and processes of using evolutionary algorithms as the search strategy to implement neural architecture search,and compares the features and development status of these neural architecture search algorithms.On this basis,this paper discusses the search space,search strategy and future development direction of neural architecture search algorithms.

Key words: neural architecture search, automated deep learning, evolutionary algorithm, search strategy, evolutionary neural network

摘要： 自动化深度学习是目前深度学习领域的研究热点，神经架构搜索算法是实现自动化深度学习的主要方法之一，该类算法可以通过对搜索空间、搜索策略或优化策略进行不同定义来自动设计神经网络结构。阐述进化算法和进化神经网络的发展历程，分类介绍以进化算法为搜索策略实现神经架构搜索的方法和过程，并比较基于进化算法的不同神经架构搜索算法的特点和现状，在此基础上，对神经架构搜索算法的搜索空间、搜索策略以及算法的未来发展方向进行探讨和展望。

关键词: 神经架构搜索, 自动化深度学习, 进化算法, 搜索策略, 进化神经网络

CLC Number:

TP391

SHANG Diya, SUN Hua, HONG Zhenhou, ZENG Qingliang. Review of Research on Neural Architecture Search Algorithms Based on Non-Gradient Evolution[J]. Computer Engineering, 2020, 46(9): 16-26.

尚迪雅, 孙华, 洪振厚, 曾庆亮. 基于无梯度进化的神经架构搜索算法研究综述[J]. 计算机工程, 2020, 46(9): 16-26.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0057520

http://www.ecice06.com/EN/Y2020/V46/I9/16

Figures/Tables 10

References

[1] WANG Jianzong,QU Xiaoyang.Dive into AutoML and AutoDL:building automated platforms for machine learning and deep learning[M].Beijing:China Machine Press,2019.(in Chinese)王健宗,瞿晓阳.深入理解AutoML和AutoDL:构建自动化机器学习与深度学习平台[M].北京:机械工业出版社,2019.
[2] QUANMING Y,MENGSHUO W,HUGO J E,et al.Taking human out of learning applications:a survey on automated machine learning[EB/OL].[2020-01-15].https://arxiv.org/abs/1810.13306v1.
[3] ZHOU Zhihua.Machine learning[M].Beijing:Tsinghua University Press,2016.(in Chinese)周志华.机器学习[M].北京:清华大学出版社,2016.
[4] JENNINGS N R,WOOLDRIDGE M J.Foundations of machine learning[M].Cambridge,USA:MIT Press,2012.
[5] ALPAYDIN E.Introduction to machine learning[M].Cambridge,USA:MIT Press,2004.
[6] FORSYTH D A,PONCE J.Computer vision:a modern approach[M].[S.l.]:Prentice Hall Professional Technical Reference,2002.
[7] GARCÍA S,LUENGO J,HERRERA F.Data reduction[M]//BENGIO Y,COURVILLE A.Intelligent systems reference library.Berlin,Germany:Springer,2014.
[8] HE Jun,GE Hong,WANG Yufeng.Overview of image segmentation algorithms[J].Computer Engineering and Science,2009,31(12):58-61.(in Chinese)何俊,葛红,王玉峰.图像分割算法研究综述[J].计算机工程与科学,2009,31(12):58-61.
[9] COLLOBERT R,WESTON J,BOTTOU L,et al.Natural language processing(almost) from scratch[J].Journal of Machine Learning Research,2011,12:2493-2537.
[10] LECUN Y,BENGIO Y,HINTON G.Deep learning[J].Nature,2015,521(7553):436-444.
[11] GOODFELLOW I,BENGIO Y,COURVILLE A.Deep learning[M].Cambridge,USA:MIT press,2016.
[12] SEIDE F,LI G,CHEN X,et al.Feature engineering in context-dependent deep neural networks for conver-sational speech transcription[C]//Proceedings of 2011 IEEE Workshop on Automatic Speech Recognition & Understanding.Washington D.C.,USA:IEEE Press,2011:24-29.
[13] CLAESKENS G,HJORT N L.Model selection and model averaging[M].Cambridge,UK:Cambridge University Press,2001.
[14] KHURANA U,NARGESIAN F,SAMULOWITZ H,et al.Automating feature engineering[J].Transformation,2016,18(10):1-10.
[15] SHANG D Y,SUN H,ZENG Q L.A reinforcement-algorithm framework for automatic model selection[J].IOP Conference Series:Earth and Environmental Science,2020,440:022060.
[16] OLSON R S,BARTLEY N,URBANOWICZ R J,et al.Evaluation of a tree-based pipeline optimization tool for automating data science[C]//Proceedings of 2016 Genetic and Evolutionary Computation Conference.New York,USA:ACM Press,2016:485-492.
[17] LECUN Y,BOTTOU L,BENGIO Y,et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324.
[18] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.ImageNet classification with deep convolutional neural networks[J].Communications of the ACM,2017,60(6):84-90.
[19] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].[2020-01-25].https://arxiv.org/pdf/1409.1556.pdf.
[20] SZEGEDY C,LIU W,JIA Y Q,et al.Going deeper with convolutions[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2015:1-9.
[21] SZEGEDY C,IOFFE S,VANHOUCKE V,et al.Inception-v4,inception-ResNet and the impact of residual connections on learning[EB/OL].[2020-01-25].https://arxiv.org/abs/1602.07261.
[22] HE K M,ZHANG X Y,REN S Q,et al.Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2016:770-778.
[23] ELSKEN T,METZEN J H,HUTTER F.Neural architecture search[EB/OL].[2020-01-25].https://arxiv.org/abs/1808.05377.
[24] BAKER B,GUPTA O,NAIK N,et al.Designing neural network architectures using reinforcement learning[EB/OL].[2020-01-25].https://arxiv.org/abs/1611.02167.
[25] ZOPH B,LE Q.Neural architecture search with reinforcement learning[EB/OL].[2020-01-25].https://arxiv.org/abs/1611.01578.
[26] SHIN R,PACKER C,SONG D.Differentiable neural network architecture search[EB/OL].[2020-01-25].https://openreview.net/pdf?id=BJ-MRKkwG.
[27] LIU H X,KAREN S,YIMING Y.Darts:differentiable architecture search[EB/OL].[2020-01-25].https://arxiv.org/abs/1806.09055.
[28] HOLLAND J H.Genetic algorithms and adaptation[M].Berlin,Germany:Springer,1984.
[29] BEYER H G,SCHWEFEL H.Evolution strategies-a comprehensive introduction[J].Natural Computing,2002,1(1):3-52.
[30] FOGEL D B,FOGEL L J,ATMAR J W.Meta-evolutionary programming[EB/OL].[2020-01-25].https://ieeexplore.ieee.org/document/186507.
[31] KOZA J R,POLI R.Genetic programming[M].Berlin,Germany:Springer,2005.
[32] GOMEZ F J,MIIKKULAINEN R.Active guidance for a finless rocket using neuro evolution[M].Berlin,Germany:Springer,2003.
[33] STANLEY K O,MIIKKULAINEN R.Evolving neural networks through augmenting topologies[J].Evolutionary Computation,2002,10(2):99-127.
[34] MIIKKULAINEN R,LIANG J,MEYERSON E,et al.Evolving deep neural networks[M].[S.l.]:Elsevier,2019.
[35] XIE L X,YUILLE A.Genetic CNN[C]//Proceedings of 2017 IEEE International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2017:1379-1388.
[36] MILLER J F,HARDING S L.Cartesian genetic programming[C]//Proceedings of GECCO'09.New York,USA:ACM Press,2009:15-26.
[37] SUGANUMA M,SHIRAKAWA S,NAGAO T.A genetic programming approach to designing convolutional neural network architectures[C]//Proceedings of Genetic and Evolutionary Computation Conference.Berlin,Germany:Springer,2017:497-504.
[38] REAL E,MOORE S,SELLE A,et al.Large-scale evolution of image classifiers[C]//Proceedings of the 34th International Conference on Machine Learning.Washington D.C.,USA:IEEE Press,2017:56-89.
[39] ZHANG H L,KIRANYAZ S,GABBOUJ M.Finding better topologies for deep convolutional neural networks by evolution[EB/OL].[2020-01-25].https://arxiv.org/abs/1809.03242.
[40] LIU H X,SIMONYAN K,VINYALS O,et al.Hierarchical representations for efficient architecture search[EB/OL].[2020-01-25].https://arxiv.org/abs/1711.00436.

Please choose a citation manager

Content to export