基于谱归一化条件生成对抗网络的图像修复算法

doi:10.19678/j.issn.1000-3428.0056287

摘要/Abstract

摘要： 基于生成对抗网络的图像修复算法在修复大尺寸缺失图像时，存在图像失真较多与判别网络性能不可控等问题，基于谱归一化条件生成对抗网络，提出一种新的图像修复算法。引入谱归一化来约束判别网络的判别性能，间接提高修复网络的修复能力，并根据控制判别网络性能对谱归一化进行理论分析。通过类别信息约束特征生成，保证修复图像的内容不变性，引入扩展卷积算子对待修复图像进行像素级操作，解决修复图像缺乏局部一致性的问题。在此基础上，运用PSNR、SSIM等图像评价方法及分片Wasserstein距离、Inception分数、流形距离度量、GAN-train和GAN-test等流形结构相似度评价指标对修复图像进行综合评价。实验结果表明，与CE、GL等算法相比，该算法获得的修复图像在主观感受和客观评价指标上均有明显提高。

关键词: 谱归一化, 条件生成对抗网络, 图像修复, 判别性能, 图像评价

Abstract: To solve the problem of large image distortion and uncontrollable discriminative network performance in image inpainting based on Generative Adversarial Network(GAN),this paper proposes a new image inpainting algorithm based on conditional generative adversarial network with spectral normalization. Spectral normalization is introduced to constrain the discriminative performance of discriminative network,and thus bring an improvement to the inpainting network performance followed by detailed theoretical analysis of spectral normalization in controlling the discriminative network performance.Category information is used to constrain feature generation to ensure that content of the inpainted image is close to that of the original image.The extended convolution operator is also introduced to perform pixel-level operation on the to-be-inpainted image to address the lack of local consistency in image inpainting. On this basis,PSNR,SSIM and other image evaluation methods,as well as slice Wasserstein distance,Inception score, manifold distance measurement,GAN-train,GAN-test and other manifold structure similarity evaluation indicators are used to comprehensively evaluate the inpainted image.Experimental results show that compared with CE,GL and other algorithms,the proposed algorithm can significantly improve the subjective and objective evaluation indicators of the inpainted images.

Key words: spectral normalization, conditional Generative Adversarial Network(GAN), image inpainting, discrimination performance, image evaluation

中图分类号:

TP391

雷蕾, 郭东恩, 靳峰. 基于谱归一化条件生成对抗网络的图像修复算法[J]. 计算机工程, 2021, 47(1): 230-238.

LEI Lei, GUO Dongen, JIN Feng. Image Inpainting Algorithm Based on Conditional Generative Adversarial Network with Spectral Normalization[J]. Computer Engineering, 2021, 47(1): 230-238.

http://www.ecice06.com/CN/Y2021/V47/I1/230

图/表 12

20210125164545

20210125164549

20210125164552

20210125164556

20210125164559

20210125164603

20210125164607

20210125164617

20210125164627

20210125164631

20210125164635

20210125164639

参考文献

[1] ZHANG Yujin.Image engineering in China:2018[J].Journal of Image and Graphics,2018,24(5):665-676.(in Chinese)章毓晋.中国图像工程:2018[J].中国图象图形学报,2018,24(5):665-676.
[2] SHEN T F,CHAN J H.Mathematical models for local non-texture inpaintings[J].SIAM Journal on Applied Mathematics,2002,62(3):1019-1043.
[3] AFONSO M,BIOUCASDIAS J,FIGUEIREDO M.An augmented lagrangian approach to the constrained optimization formulation of imaging inverse problems[J].IEEE Transactions on Image Processing,2011,20(3):681-695.
[4] ZHAO Zhengkang,LIU Ningzhong,LI Wei.Image inpainting algorithm based on synthetic sample block by gray entropy[J].Computer Engineering,2016,42(11):267-271.(in Chinese)赵政康,刘宁钟,李伟.基于灰度熵合成样本块的图像修复算法[J].计算机工程,2016,42(11):267-271.
[5] BATOOLE N,CHELLAPPA R.Detection and inpainting of facial wrinkles using texture orientation fields and markov random field modeling[J].IEEE Transactions on Image Processing,2014,34(9):3773-3788.
[6] GUO Qian,GAO Shanshan,ZHANG Xiaofeng,et al.Patch-based image inpainting via two-stage low rank approximation[J].IEEE Transactions on Vision Computer Graphics,2018,24(6):2023-2036.
[7] JIANG Fan,LIU Hui,WANG Bin,et al.Image recognition based on CNN-GRNN model[J].Computer Engineering,2017,43(4):257-262.(in Chinese)江帆,刘辉,王彬,等.基于CNN-GRNN模型的图像识别[J].计算机工程,2017,43(4):257-262.
[8] SOHEI D,ELI S,CONNELLY B.Image melding:combining inconsistent images using patch-based synthesis[J].ACM Transactions on Graphics,2012,31(4):82-91.
[9] ZHU Xuan,ZHANG Xufeng,LI Qiuju,et al.Image inpainting method based on sparse decomposition[J].Computer Science,2016,43(1):294-297.(in Chinese)祝轩,张旭峰,李秋菊,等.基于稀疏分解的图像修复方法[J].计算机工程,2016,43(1):294-297.
[10] QIANG Zhenping,HE Libo,CHEN Xu,et al.Survey on deep learning image inpainting methods[J].Journal of Image and Graphics,2019,24(3):447-463.(in Chinese)强振平,何丽波,陈旭,等.深度学习图像修复方法综述[J].中国图象图形学报,2019,24(3):447-463.
[11] SHI En,LI Qian,GU Daquan,et al.Convolutional neural network model based on local feature[J].Computer Engineering,2018,44(2):282-286.(in Chinese)施恩,李骞,顾大权,等.基于局部特征的卷积神经网络模型[J].计算机工程,2018,44(2):282-286.
[12] XIE Junyuan,XU Linli,CHEN Enhong.Image denoising and inpainting with deep neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2012:341-349.
[13] YANG Chao,LU Xin,LIN Zhe,et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:4076-4084.
[14] GOODFELLOW I,ABADIE J,MIRZA M,et al.Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2014:2672-2680.
[15] PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:feature learning by inpainting[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2016:2536-2544.
[16] SATOSHI I,EDGAR S,HIROSHI I.Globally and locally consistent image completion[J].ACM Transactions on Graphics,2017,36(4):1-14.
[17] YU J,LIM Z,YANG J,et al.Generative image inpainting with contextual attention[C]//Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:5505-5514.
[18] YOSHIDA Y,MIVATO T.Spectral norm regularization for improving the generalizability of deep learning[C]//Proceedings of the 31th International Conference on Neural Information Processing Systems.Vancouver,Canada:MIT Press,2018:1539-1542.
[19] IOFFE S,SZEGEDY C.Batch normalization:accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the 28th International Conference on Machine.Lille,France:ACM Press,2015:1-11.
[20] REN J,XU L,YAN Q,et al.Shepard convolutional neural networks[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2015:901-909.
[21] LEDIG C,WANG Z H,SHI W Z,et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:105-114.
[22] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//Proceedings of the 31th International Conference on Learning Representations.San Diego,USA:ACM Press,2015:1-14.
[23] ARIJOVASKY M,BOTTOU L.Towards principled methods for training generative adversarial networks[C]//Proceedings of the 6th International Conference on Learning Representation.Vancouver,Canada:[s.n.],2018:257-273.
[24] ARJOVSKY M,CHINTALA S,BOTTOU L.Wasserstein GAN[C]//Proceedings of the 30th International Conference on Machine Learning.Sydney,Australia:[s.n.],2017:214-223.
[25] ODENA A,OLAH C,SHLENS J.Conditional image synthesis with auxiliary classifier GANs[C]//Proceeding of the 34th International Conference on Machine Learning.Sydney,Australia:[s.n.],2017:237-251.
[26] KINGMA D,BA J.Adam:a method for stochastic optimization[C]//Proceedings of the 19th International Conference on Computational Statistics.Paris,France:[s.n.],2015:1-15.
[27] GULRAJANI I,AHMED F,ARIJOVASKY M,et al.Improved training of Wasserstein GANs[C]//Proceeding of the 31th International Conference on Neural Information Processing Systems.Long Beach,USA:MIT Press,2017:294-303.
[28] LIU G,RWDA F A,SHIH K,et al.Image inpainting for irregular holes using partial convolutions[C]//Proceeding of IEEE European Conference on Computer Vision.[S.1.]:IEEE Press,2018:89-105.
[29] HEUSEL M,RAMSAUER H,UNTERTHINER T,et al.GANs trained by a two time-scale update rule converge to a Nash equilibrium[C]//Proceeding of the 30th International Conference on Neural Information Processing Systems.Long Beach,USA:MIT Press,2017:2672-2680.
[30] SALIMANS T,GOODFELLOW I,ZAREMBA W,et al.Improved techniques for training GANs[EB/OL].[2019-09-10].https://arxiv.org/abs/abs/1606.03498.
[31] KARRAS T,AILA T,LAINE S,et al.Progressive growing of GANs for improved quality,stability,and variation[C]//Proceedings of the 5th International Conference on Learning Representations.Toulon,France:[s.n.],2017:1-26.
[32] SHMELKOV K,SCHMID C,ALAHARI K.How good is my GAN?[C]//Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:3654-3668.

选择文件类型/文献管理软件名称

选择包含的内容