基于多尺度特征融合的人脸图像修复方法

doi:10.19678/j.issn.1000-3428.0060053

摘要/Abstract

摘要： 为了解决传统方法在修复受损区域较大的图像时，产生的结果过于平滑或模糊且难以重建合理结构的问题，本文提出在传统的生成对抗网络的鉴别器中引入多尺度特征融合的方法，将不同深度的特征图经过上采样后直接相加，有效地融合了浅层以及深层的信息。该方法本质上是借助高层特征把握图像的整体规律，同时利用低层特征填充图像的细节纹理，进而可以有效地将一张图像的分辨率及其语义特征相互融合。实验结果表明，在 CelebA 数据集上，本文提出的多尺度特征融合的人脸修复方法在峰值信噪比、相似性结构、L1 损失等测度上达到了最佳效果，同时也取得了理想的视觉效果。

Abstract: In order to solve the traditional method in the repair the damaged area larger images, the result of too smooth or vague and difficult to rebuild the reasonable structure of the problem, this paper puts forward a multi-scale feature fusion method in the discriminator of the traditional Generative Adversarial Networks (GANs),the introduction of the characteristics of different depth chart after sampling on direct addition, combined with the shallow and deep information effectively. In essence, this method grasps the overall law of the image with the help of high level features and fills the detail texture of the image with low level features, so that the resolution of an image and its semantic features can be effectively fused with each other. The experimental results show that the multi-scale feature fusion method proposed in this paper achieves the best results in terms of Peak Signal to Noise Ratio , Structural Similarity, L1 loss and other measures on CelebA data set, and also achieves ideal visual effects.

白宗文 , 弋婷婷 , 周美丽 , 魏嵬. 基于多尺度特征融合的人脸图像修复方法[J]. 计算机工程, doi: 10.19678/j.issn.1000-3428.0060053.

Bai Zongwen , Yi Tingting , Zhou Meili , Wei wei . Face image inpainting method based on multi-scale feature fusion[J]. Computer Engineering, doi: 10.19678/j.issn.1000-3428.0060053.

参考文献

[1] Hiten, Pethani, Kalpesh,et al. Object Remove Using Structure Oriented In Image Inpainting[J]. National Journal of System and Information Technology, 2015, 8(2):88-96.
[2] M.-Y. Liu,T. Breuel,J. Kautz. Unsupervised Image-to-Image Translation Networks[C]// Advances in Neural Information Processing Systems 30 (NIPS). Long Beach, CA, USA: the Neural Information Processing Systems Conference(NIPs), 2017:700-708.
[3] Jo Y , Park J . SC-FEGAN: Face Editing Generative Adversarial Network with User's Sketch and Color[C]// International Conference on Computer Vision(ICCV). USA: IEEE Xplore,2019: 1745-1753.
[4] Wiesler S , Richard A , Ralf Schlüter, et al. Mean-normalized stochastic gradient for large-scale deep learning[C]// IEEE International Conference on Acoustics. Florence: Speech and Signal Processing (ICASSP), 2014: 180-184.
[5] Ioffe S , Szegedy C . Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[EB/OL].( Submitted on 11 Feb 2015 (v1), last revised 2 Mar 2015 (v3))/[2020.10.10]. http://arxiv.org/abs/1502.03167.
[6] Yu T, Guo Z, Jin X, et al. Region Normalization for Image Inpainting[C]// national conference on artificial intelligence. USA: The Association for the Advancement of Artificial Intelligence(AAAI), 2020: 12733-12740.
[7] Marcelo Bertalmío, Sapiro G , Caselles V , et al. Image inpainting[C]// Proceedings of the 27th annual conference on Computer graphics and interactive techniques. New York, USA: ACM, 2000: 417-424.
[8] Bertalmio M, Vese L A, Sapiro G, et al. Simultaneous structure and texture image inpainting[J]. IEEE Transactions on Image Processing, 2003, 12(8): 882-889.
[9] Criminisi A, Perez P, Toyama K, et al. Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transactions on Image Processing, 2004, 13(9): 1200-1212.
[10] Sun J, Yuan L, Jia J, et al. Image completion with structure propagation[C]// International Conference on Computer Graphics and Interactive Techniques. NY,USA: Association for Computing Machinery ,2005: 861-868.
[11] Barnes C, Shechtman E, Finkelstein A, et al. PatchMatch: a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3):24.
[12] Huang J, Kang S B, Ahuja N, et al. Image completion using planar structure guidance[C] //International Conference on Computer Graphics and Interactive Techniques. UAS: ACM Transactions on Graphics, 2014: 1-10.
[13] Pathak D, Krahenbuhl P, Donahue J, et al. Context Encoders: Feature Learning by Inpainting[C]//computer vision and pattern recognition(CVPR), USA: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 2536-2544.
[14] Iizuka S, Simoserra E, Ishikawa H, et al. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4):1-14.
[15] Li Y, Liu S, Yang J, et al. Generative Face Completion[C]// Computer Vision and Pattern Recognition(CVPR). USA: IEEE Xplore, 2017:3911-3919.
[16] Yang J, Qi Z, Shi Y, et al. Learning to Incorporate Structure Knowledge for Image Inpainting[C]// national conference on artificial intelligence. USA: The Association for the Advancement of Artificial Intelligence(AAAI), 2020: 12605-12612.
[17] Wang N, Li J, Zhang L, et al. MUSICAL: Multi-Scale Image Contextual Attention Learning for Inpainting[C]// International Joint Conference on Artificial Intelligence. Macao, china: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19) , 2019: 3748-3754.
[18] Xie C, Liu S, Li C, et al. Image Inpainting With Learnable Bidirectional Attention Maps[C]// International Conference on Computer Vision (ICCV). USA: IEEE Xplore , 2019: 8858-8867.
[19] Liu H, Jiang B, Xiao Y, et al. Coherent Semantic Attention for Image Inpainting[C]// International Conference on Computer Vision (ICCV). USA: Proceedings of the IEEE/CVF International Conference on Computer Vision , 2019: 4170-4179.
[20] Goodfellow, J. Pouget-Abadie, M. Mirza, et al. Generativeadversarial nets[C]// Conference and Workshop on Neural Information Processing Systems 27 (NIPs). USA: the Neural Information Processing Systems Conference(NIPs), 2014:2672–2680.
[21] Lin T Y , Dollár, Piotr, Girshick R , et al. Feature Pyramid Networks for Object Detection[C] // Computer vision and pattern recognition (CVPR). USA: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017:2117-2125.
[22] P Isola, J-Y Zhu, et al. Image-to-image translation with conditional adversarial networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).USA: IEEE Press, 2017:1125-1134.
[23] Zhu J Y , Park T , Isola P , et al. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks[C]// International Conference on Computer Vision(ICCV), USA: Proceedings of the IEEE International Conference on Computer Vision, 2017: 2223-2232.
[24] He T , Zhang Z , Zhang H , et al. Bag of Tricks for Image Classification with Convolutional Neural Networks[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: IEEE Press, 2019:558-567.
[25] Gatys L A , Ecker A S , Bethge M . Image Style Transfer Using Convolutional Neural Networks[C]// Computer Vision and Pattern Recognition(CVPR). Las Vegas, Nevada: IEEE, 2016:2414-2423.
[26] Johnson J , Alahi A , Fei-Fei L . Perceptual Losses for Real-Time Style Transfer and Super-Resolution[C]// European Conference on Computer Vision(ECCV). Berlin: Springer, 2016:694-711.
[27] Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition [EB/OL]. (Submitted on 4 Sep 2014 (v1), last revised 10 Apr 2015 (v6))/[2020.10.10]. https://arxiv.org/abs/1409.1556.
[28] Liu Z, Luo P, Wang X, et al. Deep Learning Face Attributes in the Wild[C]// International Conference on Computer Vision(ICCV). USA: Proceedings of the IEEE International Conference on Computer Vision, 2015: 3730-3738.
[29] Liu G , Reda F A , Shih K J , et al. Image Inpainting for Irregular Holes Using Partial Convolutions[C]// European Conference on Computer Vision(ECCV). Germany: Proceedings of the European Conference on Computer Vision, 2018:85-100.
[30] Adam Kingma D , Ba J . Adam: A Method for Stochastic Optimization[EB/OL]. (Submitted on 22 Dec 2014 (v1), last revised 30 Jan 2017 (v9))/[2020.10.10]. https://arxiv.org/abs/1412.6980.
[31] Alain Horé, Ziou D . Image Quality Metrics: PSNR vs. SSIM[C]// 2010 20th International Conference on Pattern Recognition, Istanbul: IEEE Press, 2010: 2366-2369.
[32] Wang Z , Bovik A C , Sheikh H R , et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4):600-612.