基于多尺度特征融合的人脸图像修复方法

doi:10.19678/j.issn.1000-3428.0060053

计算机工程 ›› 2021, Vol. 47 ›› Issue (5): 213-220,228. doi: 10.19678/j.issn.1000-3428.0060053

基于多尺度特征融合的人脸图像修复方法

白宗文¹, 弋婷婷¹, 周美丽¹, 魏嵬²

1. 延安大学物理与电子信息学院, 陕西延安 716000;
2. 西安理工大学计算机学院, 西安 710048

收稿日期:2020-11-18 修回日期:2021-01-10 发布日期:2020-12-17
作者简介:白宗文(1979-),男,副教授、博士,主研方向为深度学习;弋婷婷,硕士研究生;周美丽,副教授;魏嵬,副教授、博士。
基金资助:
国家自然科学基金（61761042，6194112）；陕西省自然科学基金（2020JM-556）；延安大学科研计划（CXY201909，YDZ2019-05）；陕西省能源大数据智能处理重点实验室开放基金（IPBED10，IPBED7）。

Face Image Inpainting Method Based on Multi-Scale Feature Fusion

BAI Zongwen¹, YI Tingting¹, ZHOU Meili¹, WEI Wei²

1. School of Physics and Electronic Information, Yan'an University, Yan'an, Shaanxi 716000, China;
2. School of Computer Science and Engineering, Xi'an University of Technology, Xi'an 710048, China

Received:2020-11-18 Revised:2021-01-10 Published:2020-12-17

摘要/Abstract

摘要： 传统图像修复方法在修复受损区域较大的图像时会出现修复结果过于平滑或模糊的现象，并且较难重建合理的人脸图像结构。在传统生成对抗网络的鉴别器中引入多尺度特征融合方法，将不同深度的特征图经过上采样后直接相加，使浅层信息和深层信息有效结合。通过借助高层特征把握图像的整体规律，同时利用低层特征填充人脸图像的细节纹理，进而使一张图像的分辨率及其语义特征相互融合，实现有效的人脸图像修复。在CelebA数据集上的实验结果表明，该方法的峰值信噪比、相似性结构、L1损失指标均优于区域归一化方法，取得了较好的视觉效果。

关键词: 图像修复, 生成对抗网络, 多尺度特征融合, 鉴别器, 高层特征, 低层特征

Abstract: When dealing with severely corrupted images,the traditional image inpainting methods often produce blurry or too smooth regions in the restored images,and have difficulty in reconstructing a reasonable face image structure.To address the problem,this paper introduces a multi-scale feature fusion method into the discriminator of the traditional Generative Adversarial Network(GAN),which directly adds the upsampled feature maps of different depth to achieve effective fusion of shallow and deep information.Then this method grasps the overall pattern of the image with the help of high-level features,and fills the detail texture of the face image with low-level features,so that the resolution of the image can be fused with its semantic features for effective face image restoration.The experimental results on the dataset of CelebA show that the proposed method outperforms the regional normalization method in terms of Peak Signal to Noise Ratio(PSNR),structural similarity,L1 loss indicators,achieving ideal visual effects.

Key words: image inpainting, Generative Adversarial Network(GAN), multi-scale feature fusion, discriminator, high-level feature, low-level feature

中图分类号:

TP391

白宗文, 弋婷婷, 周美丽, 魏嵬. 基于多尺度特征融合的人脸图像修复方法[J]. 计算机工程, 2021, 47(5): 213-220,228.

BAI Zongwen, YI Tingting, ZHOU Meili, WEI Wei. Face Image Inpainting Method Based on Multi-Scale Feature Fusion[J]. Computer Engineering, 2021, 47(5): 213-220,228.

http://www.ecice06.com/CN/Y2021/V47/I5/213

参考文献

[1] PATEL H,PETHANI H,PIPALIYA K,et al.Object remove using structure oriented in image inpainting[J].National Journal of System and Information Technology,2015,8(2):88-96.
[2] LIU M Y,BREUEL T,KAUTZ J.Unsupervised image-to-image translation networks[C]//Proceedings of Neural Information Processing Systems Conference.Long Beach,USA:[s.n.],2017:700-708.
[3] JO Y,PARK J.SC-FEGAN:face editing generative adversarial network with user's sketch and color[C]//Proceedings of International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2019:1745-1753.
[4] WIESLER S,RICHARD A,RALF S,et al.Mean-normalized stochastic gradient for large-scale deep learning[C]//Proceedings of 2014 IEEE International Conference on Acoustics.Washington D.C.,USA:IEEE Press,2014:180-184.
[5] IOFFE S,SZEGEDY C.Batch normalization:accelerating deep network training by reducing internal covariate shift[EB/OL].(2015-03-02)[2020-10-10].http://arxiv.org/abs/1502.03167.
[6] YU Tao,GUO Zongyu,JIN Xin,et al.Region normalization for image inpainting[C]//Proceedings of International Conference on Artificial Intelligence.[S.l.]:AAAI,2020:12733-12740.
[7] BERTALMÍO M,SAPIRO G,CASELLES V,et al.Image inpainting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.New York,USA:ACM Press,2000:417-424.
[8] BERTALMIO M,VESE L A,SAPIRO G,et al.Simultaneous structure and texture image inpainting[J].IEEE Transactions on Image Processing,2003,12(8):882-889.
[9] CRIMINISI A,PEREZ P,TOYAMA K,et al.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200-1212.
[10] SUN Jian,YUAN Li,JIA Jiaya,et al.Image completion with structure propagation[C]//Proceedings of International Conference on Computer Graphics and Interactive Techniques.New York,USA:ACM Press,2005:861-868.
[11] BARNES C,SHECHTMAN E,FINKELSTEIN A,et al.PatchMatch:a randomized correspondence algorithm for structural image editing[J].ACM Transactions on Graphics,2009,28(3):1-11.
[12] HUANG J,KANG S B,AHUJA N,et al.Image completion using planar structure guidance[C]//Proceedings of International Conference on Computer Graphics and Interactive Techniques.New York,USA:ACM Press,2014:1-10.
[13] PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:feature learning by inpainting[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2016:2536-2544.
[14] ⅡZUKA S,SIMOSERRA E,ISHIKAWA H,et al.Globally and locally consistent image completion[J].ACM Transactions on Graphics,2017,36(4):1-14.
[15] LI Yijun,LIU Sifei,YANG Jimei,et al.Generative face completion[C]//Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:3911-3919.
[16] YANG Jie,QI Zhiquan,SHI Yong,et al.Learning to incorporate structure knowledge for image inpainting[C]//Proceedings of International Conference on Artificial Intelligence.[S.l.]:AAAI,2020:12605-12612.
[17] WANG Ning,LI Jingyuan,ZHANG Lefei,et al.MUSICAL:multi-scale image contextual attention learning for inpainting[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence.Macao,China:[s.n.],2019:3748-3754.
[18] XIE Chaohao,LIU Shaohui,LI Chao,et al.Image inpainting with learnable bidirectional attention maps[C]//Proceedings of International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2019:8858-8867.
[19] LIU Hongyu,JIANG Bin,XIAO Yi,et al.Coherent semantic attention for image inpainting[C]//Proceedings of IEEE/CVF International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2019:4170-4179.
[20] GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C]//Proceedings of Conference and Workshop on Neural Information Processing Systems.New York,USA:ACM Press,2014:2672-2680.
[21] LIN T Y,DOLLÁR P,GIRSHICK R,et al.Feature pyramid networks for object detection[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:2117-2125.
[22] ISOLA P,ZHU J Y,ZHOU T,et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:1125-1134.
[23] ZHU J Y,PARK T,ISOLA P,et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2017:2223-2232.
[24] HE Tong,ZHANG Zhi,ZHANG Hang,et al.Bag of tricks for image classification with convolutional neural networks[C]//Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2019:558-567.
[25] GATYS L A,ECKER A S,BETHGE M.Image style transfer using convolutional neural networks[C]//Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2016:2414-2423.
[26] JOHNSON J,ALAHI A,FEI F L.Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of 2016 European Conference on Computer Vision.Berlin,Germany:Springer,2016:694-711.
[27] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].(2015-04-10)[2020-10-10].https://arxiv.org/abs/1409.1556.
[28] LIU Ziwei,LUO Ping,WANG Xiaogang,et al.Deep learning face attributes in the wild[C]//Proceedings of International Conference on Computer Vision.New York,USA:ACM Press,2015:3730-3738.
[29] LIU G,REDA F A,SHIH K J,et al.Image inpainting for irregular holes using partial convolutions[C]//Proceedings of 2018 European Conference on Computer Vision.Berlin,Germany:Springer,2018:85-100.
[30] ADAM K D,BA J.Adam:a method for stochastic optimization[EB/OL].(2017-01-30)[2020-10-10].https://arxiv.org/abs/1412.6980.
[31] ALAIN H,ZIOU D.Image quality metrics:PSNR vs.SSIM[C]//Proceeding of the 20th International Conference on Pattern Recognition.Washington D.C.,USA:IEEE Press,2010:2366-2369.
[32] WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.

选择文件类型/文献管理软件名称

选择包含的内容

基于多尺度特征融合的人脸图像修复方法

Face Image Inpainting Method Based on Multi-Scale Feature Fusion

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	宋志娜, 李莎, 杨建明, 徐川. 基于特征与区域定位增强的遥感舰船目标检测[J]. 计算机工程, 2023, 49(8): 257-264.
[2]	樊瑶, 石英男, 柏劲咸. 基于边缘与注意力跨层转移的图像修复模型[J]. 计算机工程, 2023, 49(6): 180-192.
[3]	沈梦强, 于文年, 易黎, 宋南. 基于GAN的全时间尺度语音增强方法[J]. 计算机工程, 2023, 49(6): 115-122,130.
[4]	罗嗣卿, 陈慧. 基于生成对抗网络的图像场景转换[J]. 计算机工程, 2023, 49(4): 217-225.
[5]	李培育, 张雅丽. 基于改进SRGAN模型的人脸图像超分辨率重建[J]. 计算机工程, 2023, 49(4): 199-205.
[6]	翟社平, 张宇航, 柏晓夏. 融合实体邻域信息的知识图谱嵌入负采样方法[J]. 计算机工程, 2023, 49(3): 95-104.
[7]	任家豪, 张光华, 乔钢柱, 武秀萍. 多尺度特征融合的头影标志点检测[J]. 计算机工程, 2023, 49(3): 271-279.
[8]	席荣康, 蔡满春, 芦天亮. 基于数据增强与流数据处理的Tor流量分析模型[J]. 计算机工程, 2023, 49(3): 177-184.
[9]	于敏, 屈丹, 司念文. 改进的RetinaNet目标检测算法[J]. 计算机工程, 2022, 48(8): 249-257.
[10]	赫晓慧, 宋定君, 李盼乐, 田智慧, 周广胜. 融合多尺度特征的遥感影像道路提取方法[J]. 计算机工程, 2022, 48(8): 196-205.
[11]	李晨, 侯进, 李金彪, 陈子锐. 基于注意力与残差级联的红外与可见光图像融合方法[J]. 计算机工程, 2022, 48(7): 234-240.
[12]	丁泳钧, 黄山. 一种用于图像去雾的改进生成对抗网络[J]. 计算机工程, 2022, 48(6): 207-212.
[13]	徐润昊, 程吉祥, 李志丹, 付小龙. 基于循环生成对抗网络的含遮挡人脸识别[J]. 计算机工程, 2022, 48(5): 289-296,305.
[14]	何涛, 俞舒曼, 徐鹤. 基于条件生成对抗网络与知识蒸馏的单幅图像去雾方法[J]. 计算机工程, 2022, 48(4): 165-172.
[15]	鲁统伟, 徐子昕, 闵锋. 基于生成对抗网络的知识蒸馏数据增强[J]. 计算机工程, 2022, 48(4): 70-80.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于多尺度特征融合的人脸图像修复方法

Face Image Inpainting Method Based on Multi-Scale Feature Fusion

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价