生成式对抗网络研究与应用进展

doi:10.19678/j.issn.1000-3428.0051964

摘要/Abstract

摘要： 基于零和博弈思想的生成式对抗网络（GAN）可通过无监督学习获得数据的分布，并生成较逼真的数据。基于GAN的基础概念及理论框架，研究各类GAN模型及其在特定领域的应用情况，从数据相似性度量、模型框架、训练方法3个方面进行分析，对GAN改进与扩展的相关研究成果进行总结，并从图像合成、风格迁移等应用领域展开讨论，归纳出GAN的优势与不足，同时对其应用前景进行展望。分析结果表明，GAN的学习能力与可塑性强，改进潜力大，应用范围广，但其发展面临的挑战是训练过程不稳定，且缺乏生成数据质量的客观评价标准。

关键词: 生成式对抗网络, 生成式模型, 对抗学习, 深度学习, 人工智能

Abstract: The Generative Adversarial Networks(GAN) based on the zero-sum game idea can obtain the distribution of data through unsupervised learning and generate more realistic data.Based on the basic concepts and theoretical framework of the generated confrontation network,the GAN models and the application results in specific fields are studied,and the data similarity measure,model framework and training method are summarized.The research and related research results of the improvement and expansion are analyzed,and the practical application fields such as image synthesis and style migration are discussed.The advantages and disadvantages of GAN are summarized,and the application prospects are prospected.Analysis results show that the GAN has strong learning ability and plasticity,great potential for improvement and wide application range.However,its development challenges are unstable training process and lack of objective evaluation criteria for generating data quality.

Key words: Generative Adversarial Networks(GAN), generative model, adversarial learning, deep learning, artificial intelligence

中图分类号:

TP183

柴梦婷, 朱远平. 生成式对抗网络研究与应用进展[J]. 计算机工程, 2019, 45(9): 222-234.

CHAI Mengting, ZHU Yuanping. Research and Application Progress of Generative Adversarial Networks[J]. Computer Engineering, 2019, 45(9): 222-234.

https://www.ecice06.com/CN/Y2019/V45/I9/222

图/表 6

20190912185219

20190912185222

20190912185224

20190912185227

20190912185230

20190912185233

参考文献

[1] HINTON G E,SALAKHUTDINOV R R.Reducing the dimensionality of data with neural networks[J].Science,2006,313(5786):504-507.
[2] BENGIO Y.Learning deep architectures for AI[J].Foundations and Trends in Machine Learning,2009,2(1):1-127.
[3] SCHMIDHUBER J.Deep learning in neural networks:an overview[J].Neural Networks,2015,61:85-117.
[4] HINTON G,DENG L,YU D,et al.Deep neural networks for acoustic modeling in speech recognition:the shared views of four research groups[J].IEEE Signal Processing Magazine,2012,29(6):82-97.
[5] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenet classification with deep convolutional neural networks[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2012:1097-1105.
[6] GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2014:2672-2680.
[7] KINGMA D P,WELLING M.Auto-encoding variational bayes[EB/OL].[2018-05-22].https://arxiv.org/pdf/1312.6114.pdf.
[8] 王坤峰,苟超,段艳杰,等.生成式对抗网络GAN的研究进展与展望[J].自动化学报,2017,43(3):321-332.
[9] ARJOVSKY M,CHINTALA S,BOTTOU L.Wasserstein GAN[EB/OL].[2018-05-22].https://arxiv.org/pdf/1701.07875.pdf.
[10] FUGLEDE B,TOPSOE F.Jensen-Shannon divergence and Hilbert space embedding[C]//Proceedings of International Symposium on Information Theory.Washington D.C.,USA:IEEE Press,2004:31.
[11] GULRAJANI I,AHMED F,ARJOVSKY M,et al.Improved training of wasserstein GANs[EB/OL].[2018-05-22].https://arxiv.org/pdf/1704.00028.pdf.
[12] NOWOZIN S,CSEKE B,TOMIOKA R.f-GAN:training generative neural samplers using variational divergence minimization[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2016:271-279.
[13] MAO Xudong,LI Qing,XIE Haoran,et al.Least squares generative adversarial networks[EB/OL].[2018-05-22].https://arxiv.org/pdf/1611.04076.pdf.
[14] QI Guojun.Loss-sensitive generative adversarial networks on lipschitz densities[EB/OL].[2018-05-22].https://arxiv.org/pdf/1701.06264.pdf.
[15] ZHAO Junbo,MATHIEU M,LECUN Y.Energy-based generative adversarial network[EB/OL].[2018-05-22].https://arxiv.org/pdf/1609.03126.pdf.
[16] DAI Zihang,ALMAHAIRI A,BACHMAN P,et al.Calibrating energy-based generative adversarial networks[EB/OL].[2018-05-22].https://arxiv.org/pdf/1702.01691.pdf.
[17] WANG Dilin,LIU Qiang.Learning to draw samples:with application to amortized mle for generative adversarial learning[EB/OL].[2018-05-22].https://arxiv.org/pdf/1611.01722.pdf.
[18] BERTHELOT D,SCHUMM T,METZ L.BEGAN:boundary equilibrium generative adversarial networks[EB/OL].[2018-05-22].https://arxiv.org/pdf/1703.10717.pdf.
[19] CHEN Xi,DUAN Yan,HOUTHOOFT R,et al.InfoGAN:interpretable representation learning by information maximizing generative adversarial nets[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2016:2172-2180.
[20] MAKHZANI A,FREY B J.PixelGAN autoencoders[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2017:1972-1982.
[21] MIRZA M,OSINDERO S.Conditional generative adversarial nets[EB/OL].[2018-05-22].https://arxiv.org/pdf/1411.1784.pdf.
[22] DENG Zhijie,ZHANG Hao,LIANG Xiaodan,et al.Structured generative adversarial networks[C]//Proceedings of International Conference in Neural Information Processing Systems.[S.l.]:Curran Associates Inc.,2017:3902-3912.
[23] SPRINGENBERGJ T.Unsupervised and semi-supervised learning with categorical generative adversarial networks[EB/OL].[2018-05-22].https://arxiv.org/pdf/1511.06390.pdf.
[24] ZHANG Han,XU Tao,LI Hongsheng,et al.StackGAN:text to photo-realistic image synthesis with stacked generative adversarial networks[EB/OL].[2018-05-22].https://arxiv.org/pdf/1612.03242.pdf.
[25] RADFORD A,METZ L,CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[EB/OL].[2018-05-22].https://arxiv.org/pdf/1511.06434.pdf.
[26] LECUN Y,BOTTOU L,BENGIO Y,et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324.
[27] LECUN Y,BOSER B,DENKER J S,et al.Backpropagation applied to handwritten zip code recognition[J].Neural Computation,1989,1(4):541-551.
[28] YI Zili,ZHANG Hao,PING Tan,et al.DualGAN:unsupervised dual learning for image-to-image translation[EB/OL].[2018-05-22]. https://arxiv.org/pdf/1704.02510.pdf.
[29] KIM T,CHA M,KIM H,et al.Learning to discover cross-domain relations with generative adversarial networks[C]//Proceedings of the 34th International Conference on Machine Learning.Berlin,Germany:Springer,2017:1857-1865.
[30] ZHU Junyan,PARK T,ISOLA P,et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[EB/OL].[2018-05-22].https://arxiv.org/pdf/1703.10593.pdf.
[31] GAN Zhe,CHEN Liqun,WANG Weiyao,et al.Triangle generative adversarial networks[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2017:5253-5262.
[32] NGUYEN T,LE T,VU H,et al.Dual discriminator generative adversarial nets[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press, 2017:2667-2677.
[33] LI Chongxuan,XU Tun,ZHU Jun,et al.Triple generative adversarial nets[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press, 2017:4091-4101.
[34] CHAVDAROVA T,FLEURET F.SGAN:an alternative training of generative adversarial networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:9407-9415
[35] DENTON E,CHINTALA S,FERGUS R.Deep generative image models using a Laplacian pyramid of adversarial networks[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2015:1486-1494.
[36] IM D J,KIM C D,JIANG Hui,et al.Generating images with recurrent adversarial networks[EB/OL].[2018-05-22].https://arxiv.org/pdf/1602.05110.pdf.
[37] GREGOR K,DANIHELKA I,GRAVES A,et al.DRAW:a recurrent neural network for image generation[C]//Proceedings of International Conference on International Conference on Machine Learning.[S.l.]:JMLR.org,2015:1462-1471.
[38] SALIMANS T,GOODFELLOW I,ZAREMBA W,et al.Improved techniques for training GANs[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2016:2234-2242.
[39] LARSEN A B L,SØNDERBY S K,LAROCHELLE H,et al.Autoencoding beyond pixels using a learned similarity metric[C]//Proceedings of International Conference on Machine Learning.[S.l.]:JMLR.org,2016:1558-1566.
[40] MESCHEDER L,NOWOZIN S,GEIGER A.Adversarial variational Bayes:unifying variational autoencoders and generative adversarial networks[C]//Proceedings of International Conference on International Conference on Machine Learning.[S.l.]:JMLR.org,2017:2391-2400.
[41] SALIMANS T,KARPATHY A,CHEN Xi,et al.PixelCNN++:improving the pixelCNN with discretized logistic mixture likelihood and other modifications[EB/OL].[2018-05-22]. https://arxiv.org/pdf/1701.05517.pdf.
[42] OORDA V D,KALCHBRENNER N,KAVUKCUOGLU K.Pixel recurrent neural networks[C]//Proceedings of International Conference on International Conference on Machine Learning.[S.l.]:JMLR.org,2016:1747-1756.
[43] DOSOVITSKIY A,BROX T.Generating images with perceptual similarity metrics based on deep networks[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2016:658-666.
[44] GRAVES A.Generating sequences with recurrent neural networks[EB/OL].[2018-05-22].https://arxiv.org/pdf/1308.0850.pdf.
[45] LEE H Y,TSENG B H,WEN T H,et al.Personalizing recurrent-neural-network-based language model by social network[J].IEEE/ACM Transactions on Audio,Speech,and Language Processing,2017,25(3):519-530.
[46] MOGRENO.C-RNN-GAN:continuous recurrent neural networks with adversarial training[EB/OL].[2018-05-22].https://arxiv.org/pdf/1611.09904.pdf.
[47] YU Lantao,ZHANG Weinan,WANG Jun,et al.SeqGAN:sequence generative adversarial nets with policy gradient[C]//Proceedings of Conference on Artificial Intelligence.[S.l.]:AAAI Press,2017:2852-2858.
[48] HOCHREITER S,SCHMIDHUBER J.Long short-term memory[J].Neural Computation,1997,9(8):1735-1780.
[49] SAK H,SENIOR A,BEAUFAYS F.Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition[EB/OL].[2018-05-22]. https://arxiv.org/pdf/1402.1128v1.pdf.
[50] KWAK H,ZHANG B T.Generating images part by part with composite generative adversarial networks[EB/OL].[2018-05-22].https://arxiv.org/pdf/1607.05387.pdf.
[51] YANG Jianwei,KANNAN A,BATRA D,et al.LR-GAN:Layered recursive generative adversarial networks for image generation[EB/OL].[2018-05-22].https://arxiv.org/pdf/1703.01560.pdf.
[52] GATYS L A,ECKER A S,BETHGE M.A neural algorithm of artistic style[EB/OL].[2018-05-22].https://arxiv.org/pdf/1508.06576.pdf.
[53] JOHNSON J,ALAHI A,LI Feifei.Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of European Conference on Computer Vision.Berlin,Germany:Springer,2016:694-711.
[54] BOUSMALIS K,SILBERMAN N,DOHAN D,et al.Unsupervised pixel-level domain adaptation with generative adversarial networks[EB/OL].[2018-05-22].https://arxiv.org/pdf/1612.05424.pdf.
[55] LIU Mingyu,BREUEL T,KAUTZ J.Unsupervised image-to-image translation networks[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2017:700-708.
[56] SHRIVASTAVA A,PFISTER T,TUZEL O,et al.Learning from simulated and unsupervised images through adversarial training[EB/OL].[2018-05-22].https://arxiv.org/pdf/1612.07828.pdf.
[57] ISOLA P,ZHU Junyan,ZHOU Tinghui,et al.Image-to-image translation with conditional adversarial networks[EB/OL].[2018-05-22]. https://arxiv.org/pdf/1611.07004.pdf.
[58] LIU Mingyu,TUZEL O.Coupled generative adversarial networks[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2016:469-477.
[59] LEDIG C,THEIS L,HUSZÁR F,et al.Photo-realistic single image super-resolution using a generative adversarial network[EB/OL].[2018-05-22].https://arxiv.org/pdf/1609.04802v1.pdf.
[60] HE Kaiming,ZHANG Xiangyu,REN Shaoqing,et al.Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2016:770-778.
[61] BOER J F D,CENSE B,PARK B H,et al.Improved signal-to-noise ratio in spectral-domain compared with time-domain optical coherence tomography[J].Optics Letters,2003,28(21):2067-2069.
[62] ARDENKJÆR-LARSEN J H,FRIDLUND B,GRAM A,et al.Increase in signal-to-noise ratio of > 10,000 times in liquid-state NMR[J].Proceedings of the National Academy of Sciences,2003,100(18):10158-10163.
[63] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].[2018-05-22].https://arxiv.org/pdf/1409.1556.pdf.
[64] ZHANG Jun,YAN Yong,LADES M.Face recognition:eigenface,elastic matching,and neural nets[J].Proceedings of the IEEE,1997,85(9):1423-1435.
[65] CHOPRA S,HADSELL R,LECUN Y.Learning a similarity metric discriminatively,with application to face verification[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2005:539-546.
[66] HUANG Rui,ZHANG Shu,LI Tianyu,et al.Beyond face rotation:global and local perception GAN for photorealistic and identity preserving frontal view synthesis[EB/OL].[2018-05-22].https://arxiv.org/pdf/1704.04086.pdf.
[67] 姚乃明,郭清沛,乔逢春,等.基于生成式对抗网络的鲁棒人脸表情识别[J].自动化学报,2018,44(5):865-877.
[68] LI Yijun,LIU Sifei,YANG Jimei,et al.Generative face completion[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:3911-3919.
[69] KARPATHY A,LI Feifei.Deep visual-semantic alignments for generating image descriptions[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2015:3128-3137.
[70] REED S,AKATA Z,YAN Xinchen,et al.Generative adversarial text to image synthesis[C]//Proceedings of International Conference on International Conference on Machine Learning.[S.l.]:JMLR.org,2016:1060-1069.
[71] XU Tao,ZHANG Pengchuan,HUANG Qiuyuan,et al.AttnGAN:fine-grained text to image generation with attentional generative adversarial networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:1316-1324.
[72] CHEN Yang,LAI Yukun,LIU Yongjin.CartoonGAN:generative adversarial networks for photo cartoonization[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:9465-9474.
[73] PERARNAU G,WEIJER J V,RADUCANU B,et al.Invertible conditional GANs for image editing[EB/OL].[2018-05-22].https://arxiv.org/pdf/1611.06355.pdf.
[74] KOSSAIFI J,TRAN L,PANAGAKIS Y,et al.GAGAN:geometry-aware generative adverserial networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:878-887.
[75] MA Shuang,FU Jianlong,CHEN Changwei,et al.DA-GAN:instance-level image translation by deep attention generative adversarial networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:5657-5666.
[76] WANG Zongwei,TANG Xu,LUO Weixin,et al.Face aging with identity-preserved conditional generative adversarial networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:7939-7947.
[77] CHOI Y,CHOI M,KIM M,et al.StarGAN:unified generative adversarial networks for multi-domain image-to-image translation[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:8789-8797.
[78] REED S E,AKATA Z,MOHAN S,et al.Learning what and where to draw[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2016:217-225.
[79] BAI Yancheng,ZHANG Yongqiang,DING Mingli,et al.Finding tiny faces in the wild with generative adversarial network[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:21-30.
[80] WANG Jifeng,LI Xiang,HUI Le,et al.Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:1788-1797.
[81] QIAN Rui,TAN R T,YANG Wenhan,et al.Attentive generative adversarial network for raindrop removal from a single image[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:2482-2491.
[82] CHEN J,CHEN J,CHAO H,et al.Image blind denoising with generative adversarial network based noise modeling[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:3155-3164.
[83] LI Ruide,PAN Jinshan,LI Zechao,et al.Single image dehazing via conditional generative adversarial network[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Computer Society,2018:8202-8211.
[84] NGUYEN V,VICENTE T F Y,ZHAO Maozheng,et al.Shadow detection with conditional generative adversarial networks[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2017:4520-4528.
[85] VONDRICK C,PIRSIAVASH H,TORRALBA A.Generating videos with scene dynamics[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2016:613-621.
[86] BERNHARD K,HUANG Zhiwu,DANDA P P,et al.Improving video generation for multi-functional applications[EB/OL].[2018-05-22].https://arxiv.org/pdf/1711.11453v2.pdf.
[87] SAITO M,MATSUMOTO E,SAITO S.Temporal generative adversarial nets with singular value clipping[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:2830-2839.
[88] TULYAKOV S,LIU Mingyu,YANG Xiaodong,et al.MoCoGAN:decomposing motion and content for video generation[EB/OL].[2018-05-22].https://arxiv.org/pdf/1707.04993.pdf.
[89] 孔德江,汤斯亮,吴飞.时空嵌入式生成对抗网络的地点预测方法[J].模式识别与人工智能,2018,31(1):49-60.
[90] XIONG Wei,LUO Wenhan,MA Lin,et al.Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:2364-2373.
[91] GUPTA A,JOHNSON J,LI Feifei,et al.Social GAN:socially acceptable trajectories with generative adversarial networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:2255-2264.
[92] WANG Weiyue,HUANG Qiangui,YOU Suya,et al.Shape inpainting using 3D generative adversarial network and recurrent convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:2298-2306.
[93] WU Jiajun,ZHANG Chengkai,XUE Tianfan,et al.Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling[C]//Proceedings of International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2016:82-90.
[94] SAGE A,AGUSTSSON E,TIMOFTE R,et al.Logo synthesis and manipulation with clustered generative adversarial networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:5879-5888.
[95] DOLHANSKY B,FERRER C C.Eye in-painting with exemplar generative adversarial networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:5657-5666.
[96] HAUSMAN K,CHEBOTAR Y,SCHAAL S,et al.Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets[EB/OL].[2018-05-22].https://arxiv.org/pdf/1705.10479.pdf.
[97] BARAM N,ANSCHEL O,CASPI I,et al.End-to-end differentiable adversarial imitation learning[C]//Proceedings of International Conference on International Conference on Machine Learning.[S.l.]:JMLR.org,2017:390-399.
[98] ZHANG Yizhe,GAN Zhe,FAN Kai,et al.Adversarial feature matching for text generation[C]//Proceedings of International Conference on International Conference on Machine Learning.[S.l.]:JMLR.org,2017:4006-4015.
[99] MAKHZANI A,SHLENS J,JAITLY N,et al.Adversarial autoencoders[EB/OL].[2018-05-22].https://arxiv.org/pdf/1511.05644.pdf.

选择文件类型/文献管理软件名称

选择包含的内容