融合动态特征与注意力的敦煌壁画修复模型

doi:10.19678/j.issn.1000-3428.0067371

摘要/Abstract

摘要： 敦煌壁画图像含有丰富的纹理和结构信息,在破损壁画修复过程中,容易忽略受损特征信息与完整特征信息之间的区别,从而误导修复过程生成不合理的壁画内容。针对该问题,提出一种融合动态特征选择和像素级通道注意力的壁画修复模型。设计基于U-Net 的网络生成器,实现对破损图像的编码与解码操作;采用有效可迁移卷积模块,通过动态选择采样空间位置,实现对有效特征信息的灵活提取,采用区域综合归一化模块减少修复区域与完整区域的期望和方差的偏移,从而加强对有效特征信息的选择和利用;在解码层设计像素级通道注意力模块,在增强有效特征权重的同时使模型可从相隔较远的空间位置学习有效特征。在敦煌壁画数据集上的实验结果表明,该算法能够利用有效信息修复掩膜区域比例不一的不规则破损壁画图像,相比PConv、PRVS、DSNet算法,在峰值信噪比(PSNR)指标上平均提升 0.502 dB,在结构相似性(SSIM)指标上平均提升 1.384%。

关键词: 信息处理技术, 壁画修复, 深度学习, 有效特征选择, 注意力机制

Abstract: Dunhuang mural images contain abundant textural and structural information. During the repair of inpainting murals, differences between damaged and complete feature information are easily overlooked, which can result in unsatisfactory repair and unreasonable mural content. Hence, a mural-restoration model that integrates dynamic feature selection and Pixel-Level Channel Attention(PLCA) is proposed in this study. It designs a U-Net-based network generator to encode and decode inpainting images as well as adopts an effective transferable convolution module that dynamically selects the sampling-space position to flexibly extract effective feature information. A region synthesis-normalization module is used to reduce the expected and variance deviations between the repaired and complete areas, thereby strengthening the selection and utilization of effective feature information. Finally, a PLCA module is designed in the decoding layer to enhance the effective feature weights while allowing the model to learn effective features from distant spatial positions. Experimental results on the Dunhuang mural dataset show that the proposed algorithm can effectively use information to repair irregularly damaged mural images with varying proportions of mask areas. Compared to PConv, PRVS, DSNet algorithms, the proposed algorithm improves the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) index by 0.502 dB and 1.384% on average, respectively.

Key words: information processing technology, mural inpainting, deep learning, effective feature selection, attention mechanism

中图分类号:

TP391

刘仲民, 严梁. 融合动态特征与注意力的敦煌壁画修复模型[J]. 计算机工程, 2024, 50(5): 342-353.

LIU Zhongmin, YAN Liang. Inpainting Model of Dunhuang Mural Fusing Dynamic Feature and Attention[J]. Computer Engineering, 2024, 50(5): 342-353.

https://www.ecice06.com/CN/Y2024/V50/I5/342

参考文献

[1] BERTALMIO M, SAPIRO G, CASELLES V, et al. Image inpainting[EB/OL].[2023-03-01]. https://dl.acm.org/doi/pdf/10.1145/344779.344972.
[2] WANG H, LI Q Q, JIA S. A global and local feature weighted method for ancient murals inpainting[J]. International Journal of Machine Learning and Cybernetics,2020,11(6):1197-1216.
[3] ZHANG D, LIANG Z, YANG G, et al. A robust forgery detection algorithm for object removal by exemplar-based image inpainting[J]. Multimedia Tools and Applications,2018,77(10):11823-11842.
[4] SRIDEVI G, KUMAR S S. Image inpainting based on fractional-order nonlinear diffusion for image reconstruction[J]. Circuits, Systems, and Signal Processing,2019,38(8):3802-3817.
[5] LI P, CHEN W G, NG M K. Compressive total variation for image reconstruction and restoration[J]. Computers and Mathematics with Applications,2020,80(5):874-893.
[6] FAN Y. Damaged region filling by improved criminisi image inpainting algorithm for thangka[J]. Cluster Computing,2019,22(6):13683-13691.
[7] YANG S Y, LIANG H T, WANG Y, et al. Image inpainting based on multi-patch match with adaptive size[J]. Applied Sciences, 2020,10(14):4921.
[8] SUN J, YUAN L, JIA J Y, et al. Image completion with structure propagation[J]. ACM Transactions on Graphics, 2005, 24(3):861-868.
[9] WEI Y W, LIU S G. Domain-based structure-aware image inpainting[J]. Signal, Image and Video Processing, 2016, 10:911-919.
[10] LIU G L, REDA F A, SHIH K J, et al. Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the European Conference on Computer Vision. Berlin, Germany:Springer, 2018:85-100.
[11] YU J H, LIN Z, YANG J M, et al. Free-form image inpainting with gated convolution[EB/OL].[2023-03-01]. https://arxiv.org/abs/1806.03589.
[12] HUI Z, LI J, WANG X M, et al. Image fine-grained inpainting[EB/OL].[2023-03-01]. https://arxiv.org/abs/2002.02609v2.
[13] WANG Y, CHEN Y C, TAO X, et al. VCNet:a robust approach to blind image inpainting[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany:Springer, 2020:752-768.
[14] YU T, GUO Z Y, JIN X, et al. Region normalization for image inpainting[EB/OL].[2023-03-01]. https://arxiv.org/abs/1911.10375.
[15] YU J H, LIN Z, YANG J M, et al. Generative image inpainting with contextual attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA:IEEE Press, 2018:5505-5514.
[16] YAN Z Y, LI X M, LI M, et al. Shift-Net:image inpainting via deep feature rearrangement[C]//Proceedings of the European Conference on Computer Vision. Berlin, Germany:Springer, 2018:1-17.
[17] ZENG Y H, FU J L, CHAO H Y, et al. Learning pyramid-context encoder network for high-quality image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA:IEEE Press,2019:1486-1494.
[18] 胡凯,赵健,刘昱,等.结构引导的图像修复[J].北京航空航天大学学报, 2022,48(7):1269-1277. HU K, ZHAO J, LIU Y, et al. Images inpainting via structure guidance[J]. Journal of Beijing University of Aeronautics and Astronautics, 2022,48(7):1269-1277. (in Chinese)
[19] 陈永,陈锦,陶美风.多尺度特征和注意力融合的生成对抗壁画修复[J].北京航空航天大学学报,2023, 49(2):254-264. CHEN Y, CHEN J, TAO M F. Mural inpainting with generative adversarial networks based on multi-scale feature and attention fusion[J]. Journal of Beijing University of Aeronautics and Astronautics, 2023, 49(2):254-264. (in Chinese)
[20] 孙琪,翟锐,左方,等.基于部分卷积和多尺度特征融合的人脸图像修复模型[J].计算机工程与科学, 2023, 45(2):304-312. SUN Q, ZHAI R, ZUO F, et al. Facial image inpainting based on partial convolution and multi-scale feature integration[J]. Computer Engineering & Science, 2023, 45(2):304-312.(in Chinese)
[21] RONNEBERGER O, FISCHER P, BROX T. U-Net:convolutional networks for biomedical image segmentation[C]//Proceedings of the International Conference on Medical Image Computing and Computer-assisted Intervention. Berlin, Germany:Springer, 2015:234-241.
[22] DAI J F, QI H Z, XIONG Y W, et al. Deformable convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision. Washington D. C., USA:IEEE Press, 2017:764-773.
[23] LOFFE S, SZEGEDY C. Batch normalization:accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the 32nd International Conference on International Conference on Machine Learning. New York, USA:ACM Press, 2015:448-456.
[24] BA J L, KIROS J R, HINTON G E. Layer normalization[EB/OL].[2023-03-01]. https://arxiv.org/pdf/1607.06450.pdf.
[25] ULYANOV D, VEDALDI A, LEMPITSKY V. Instance normalization:the missing ingredient for fast stylization[EB/OL].[2023-03-01]. https://arxiv.org/pdf/1607.08022.pdf.
[26] YU T X, ZHANG S J, LIN C, et al. Dunhuang grottoes painting dataset and benchmark[EB/OL].[2023-03-01].http://arXiv preprint arXiv:1907.04589, 2019.
[27] WANG N, ZHANG Y P, ZHANG L F. Dynamic selection network for image inpainting[J]. IEEE Transactions on Image Processing, 2021, 30:1784-1798.
[28] LI J Y, HE F X, ZHANG L F, et al. Progressive reconstruction of visual structure for image inpainting[C]//Proceedings of the International Conference on Computer Vision. Washington D. C., USA:IEEE Press, 2019:5962-5971.

选择文件类型/文献管理软件名称

选择包含的内容