Research on Structure Search Method for Deep Neural Network Considering Diversity

doi:10.19678/j.issn.1000-3428.0058766

Abstract

Abstract: In order to improve the ability of network structure optimization,this paper proposes an improved deep neural network structure search method.Aiming at the problem that it is difficult to measure the distance between network structures,a graph-based method is proposed to measure the distance between deep neural network structures based on the structure search scheme of neural network.By analyzing the performance of network structure that is respectively trained by a small number of steps and fully trained,a method to find the diverse optimal network structure is proposed based on the advantages of diversity solutions.Experimental results show that the neural network structure search method can effectively improve the solution quality and help in finding a better network structure.

Key words: deep neural network, structure search, diversity, network alignment, node similarity

摘要： 为提升网络结构的寻优能力，提出一种改进的深度神经网络结构搜索方法。针对网络结构间距难以度量的问题，结合神经网络的结构搜索方案，设计基于图的深度神经网络结构间距度量方式。对少量步数训练和充分训练2种情况下的网络结构性能进行分析，基于多样性解的优势，给出一种多样性最优网络结构搜索方法。实验结果表明，该方法能够有效提高解的质量，有助于寻找到更优的网络结构。

关键词: 深度神经网络, 结构搜索, 多样性, 网络对准, 节点相似性

CLC Number:

TP183

YANG Xiaomei, GUO Wenqiang, ZHANG Juling. Research on Structure Search Method for Deep Neural Network Considering Diversity[J]. Computer Engineering, 2020, 46(12): 105-112,133.

杨晓梅, 郭文强, 张菊玲. 考虑多样性的深度神经网络结构搜索方法研究[J]. 计算机工程, 2020, 46(12): 105-112,133.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0058766

http://www.ecice06.com/EN/Y2020/V46/I12/105

Figures/Tables 9

References

[1] ZOPH B,VASUDEVAN V,SHLENS J,et al.Learning transferable architectures for scalable image recognition[EB/OL].[2020-05-20].https://openaccess.thecvf.com/content_cvpr_2018/papers/Zoph_Learning_Transferable_Architectures_CVPR_2018_paper.pdf.
[2] PHAM H,GUAN M,ZOPH B,et al.Efficient neural architecture search via parameter sharing[EB/OL].[2020-05-20].https://arxiv.org/abs/1802.03268.
[3] CAI Han,CHEN Tianyao,ZHANG Wennan,et al.Efficient architecture search by network transformation[C]//Proceed-ings of the 32nd Conference on Artificial Intelligence.Washington D.C.,USA:IEEE Press,2018:151-163.
[4] LIU C,ZOPH B,SHLENS J,et al.Progressive neural architecture search[EB/OL].[2020-05-20].https://arxiv.org/pdf/1712.00559.pdf.
[5] BAKER B,GUPTA O,NAIK N,et al.Designing neural network architectures using reinforcement learning[EB/OL].[2020-05-20].https://arxiv.org/abs/1611.02167.
[6] ZHONG Zhao,YAN Junjie,WU Wei,et al.Practical block-wise neural network architecture generation[EB/OL].[2020-05-20].https://arxiv.org/abs/1708.05552.
[7] LIU Yong,YAO Xin.A population-based learning algorithm which learns both architectures and weights of neural networks[J].Chinese Journal of Advanced Software Research,1996,3(1):54-65.
[8] PIERGIOVANNI A,ANGELOVA A,TOSHEV A,et al.Evolving space-time neural architectures for videos[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2019:16-28.
[9] DENG Ningyi,SHEN Zhiqiang,GUO Yuefei.Significance detection based on category prior and deep neural network features[J].Computer Engineering,2017,43(6):225-229.(in Chinese) 邓凝旖,沈志强,郭跃飞.基于类别先验与深度神经网络特征的显著性检测[J].计算机工程,2017,43(6):225-229.
[10] VAHDAT A,MALLYA A,LIU M Y,et al.UNAS:differentiable architecture search meets reinforcement learning[EB/OL].[2020-05-20].https://arxiv.org/abs/1912.07651.
[11] KANDASAMY K,NEISWANGER W,SCHNEIDER J,et al.Neural architecture search with Bayesian optimisa-tion and optimal transport[EB/OL].[2020-05-20].https://arxiv.org/abs/1802.07191.
[12] VOGELSTEIN J T,CONROY J M,LYZINSKI V,et al.Fast approximate quadratic programming for graph matching[J].PLoS One,2015,10(4):1-17.
[13] SINGH R,XU J,BERGER B.Global alignment of multiple protein interaction networks with application to functional orthology detection[J].Proceedings of the National Academy of Sciences of the United States of America,2008,105(35):12763-12768.
[14] LIAO C S,LU K,BAYM M,et al.IsoRankN:spectral methods for global alignment of multiple protein networks[J].Bioinformatics,2009,25(12):1253-1258.
[15] KOLLIAS G,MOHAMMADI S,GRAMA A.Network Similarity Decomposition(NSD):a fast and scalable approach to network alignment[J].IEEE Transactions on Knowledge and Data Engineering,2012,24(12):2232-2243.
[16] GLIGORIJEVI V,MALOD-DOGNIN N,PRŽULJ N.Fuse:multiple network alignment via data fusion[J].Bioinformatics,2016,32(8):1195-1203.
[17] BAYATI M,GERRITSEN M,GLEICH D F,et al.Algorithms for large,sparse network alignment problems[C]//Proceedings of 2009 IEEE International Conference on Data Mining.Washington D.C.,USA:IEEE Press,2009:705-710.
[18] KOUTRA D,TONG H H,LUBENSKY D.BIG-ALIGN:fast bipartite graph alignment[C]//Proceedings of 2013 IEEE International Conference on Data Mining.Washington D.C.,USA:IEEE Press,2013:389-398.
[19] BAYATI M,SHAH D,SHARMA M.Maximum weight matching via max-product belief propagation[J].IEEE Transactions on Information Theory,2005,54(3):1763-1767.
[20] MALMI E,CHAWLA S,GIONIS A.Lagrangian relaxations for multiple network alignment[J].Data Mining and Knowledge Discovery,2017,31(5):1331-1358.
[21] ZHANG Si,TONG Hanghang.Final:fast attributed network alignment[C]//Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.New York,USA:ACM Press,2016:1224-1237.
[22] HEIMANN M,SHEN H M,SAFAVI T,et al.REGAL:representation learning-based graph alignment[EB/OL].[2020-05-20].https://arxiv.org/abs/1802.06257.
[23] DRINEAS P,MAHONEY M W.Approximating a gram matrix for improved kernel-based learning[M].Berlin,Germany:Springer,2005.

Please choose a citation manager

Content to export