基于可变形草图引导的图像修复方法

doi:10.19678/j.issn.1000-3428.0252645

摘要/Abstract

摘要： 草图引导的图像修复技术在照片修复、创意编辑等领域具有重要的应用价值,但面临用户草图数据稀缺与几何偏差导致的修复失真双重挑战。现有方法依赖边缘检测生成伪草图,但忽略用户手绘偏差(如手抖、笔触断裂),导致在复杂场景下结构错位与细节模糊。针对上述挑战,提出一种联合可变形草图生成网络(DSN)与双阶段引导修复的创新框架。首先构建可变形草图生成网络,通过建模典型手绘偏差,生成具有真实几何变形特征的大规模草图-图像配对数据集,有效缓解数据稀缺问题;其次设计两阶段修复框架,第一阶段针对用户输入草图进行几何失准校正与结构断裂修复,实现草图优化,第二阶段将优化后的草图信息有效融入修复网络,实现全局结构约束与局部纹理生成的协同优化。在基准数据集上的实验验证了该方法的有效性,实验结果表明,在CelebA-HQ数据集上,该方法的峰值信噪比(PSNR)为25.78 dB,结构相似性(SSIM)为0.852,该方法有效解决了用户草图数据稀缺与几何偏差问题,显著提升了草图引导图像修复在结构准确性和感知质量方面的性能。

关键词: 可变形草图生成网络, 双阶段协同优化, 图像修复, 对抗生成网络, 草图-图像对生成

Abstract: Sketch-guided image inpainting has significant application value in photo restoration and creative editing. However, it faces the dual challenges of scarce user sketch data and restoration distortions caused by geometric deviations. Existing methods rely on edge detection to generate pseudo-sketches while neglecting user-drawn deviations (e.g., hand tremors and stroke breaks), leading to structural misalignment and detail blurring in complex scenes. To address these challenges, this study proposes an innovative framework that combines a Deformable Sketch generation Network (DSN) with dual-stage-guided inpainting. First, a DSN is constructed to model typical hand-drawn deviations, generating a large-scale sketch-image paired dataset with realistic geometric deformation features and effectively alleviating data scarcity. Second, a two-stage inpainting framework is designed: the first stage corrects geometric misalignment and repairs structural breaks in input sketches to optimize the sketches, whereas the second stage effectively integrates the optimized sketch information into the inpainting network to achieve collaborative optimization of global structural constraints and local texture generation. Experiments on benchmark datasets validate the effectiveness of the method, achieving a Peak Signal-to-Noise Ratio (PSNR) of 25.78 dB and Structural Similarity Index Metric (SSIM) of 0.852 on the CelebA-HQ dataset. The results demonstrate that this method effectively addresses the challenges of scarce user sketch data and geometric deviations while significantly improving the structural accuracy and perceptual quality of sketch-guided image inpainting.

Key words: Deformable Sketch generation Network (DSN), dual-stage collaborative optimization, image inpainting, Generative Adversarial Network (GAN), sketch-image paired synthesis

中图分类号:

TP391

杨红菊, 刘娜, 李尧, 曹付元. 基于可变形草图引导的图像修复方法[J]. 计算机工程, 2026, 52(10): 177-187.

YANG Hongju, LIU Na, LI Yao, CAO Fuyuan. Deformable Sketch-Guided Image Inpainting Method[J]. Computer Engineering, 2026, 52(10): 177-187.

https://www.ecice06.com/CN/Y2026/V52/I10/177

参考文献

[1] SUN J, YUAN L, JIA J Y, et al. Image completion with structure propagation[C]//Proceedings of the ACM SIGGRAPH 2005 Papers. New York, USA: ACM Press, 2005: 861-868.
[2] ZENG Y, LIN Z, PATEL V M. SketchEdit: mask-free local image manipulation with partial sketches[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2022: 5941-5951.
[3] YU J H, LIN Z, YANG J M, et al. Free-form image inpainting with gated convolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA:IEEE Press, 2019: 4470-4479.
[4] XIE S N, TU Z W. Holistically-nested edge detection[C]//Proceedings of the IEEE International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2015: 1395-1403.
[5] LIU Z W, LUO P, WANG X G, et al. Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2015: 3730-3738.
[6] ZHOU B L, LAPEDRIZA A, KHOSLA A, et al. Places: a 10 million image database for scene recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(6): 1452-1464.
[7] CRIMINISI A, PEREZ P, TOYAMA K. Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transactions on Image Processing, 2004, 13(9): 1200-1212.
[8] BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: a randomized correspondence algorithm for structural image editing[M]//Seminal graphics papers: pushing the boundaries, Volume 2. New York, USA: ACM Press, 2023: 619-629.
[9] HAYS J, EFROS A A. Scene completion using millions of photographs[J]. ACM Transactions on Graphics, 2007, 26(99): 4.
[10] 李运堂,朱文凯,李恒杰,等. 基于轻量型编解码网络的复杂输电线图像识别[J]. 光电工程,2024,51(10): 240158. LI Y T, ZHU W K, LI H J, et al. Image recognition of complex transmission lines based on lightweight encoder-decoder networks[J]. Opto-Electronic Engineering, 2024, 51(10): 240158. (in Chinese)
[11] IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1-14.
[12] YU J H, LIN Z, YANG J M, et al. Generative image inpainting with contextual attention[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2018: 5505-5514.
[13] ZHENG C X, CHAM T J, CAI J F. Pluralistic image completion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2019: 1438-1447.
[14] SUVOROV R, LOGACHEVA E, MASHIKHIN A, et al. Resolution-robust large mask inpainting with Fourier convolutions[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Washington D.C., USA: IEEE Press, 2022: 3172-3182.
[15] ZHANG R S, QUAN W Z, ZHANG Y, et al. W-Net: structure and texture interaction for image inpainting[J]. IEEE Transactions on Multimedia, 2023, 25: 7299-7310.
[16] LUGMAYR A, DANELLJAN M, ROMERO A, et al. RePaint: inpainting using denoising diffusion probabilistic models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 11451-11461.
[17] ZHANG L S, CHEN Q C, HU B T, et al. Text-guided neural image inpainting[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York, USA: ACM Press, 2020: 1302-1310.
[18] 杨红菊, 翟艳峰. 基于D-ResNeXt骨干网络的小样本图像分类算法[J]. 山西大学学报(自然科学版), 2024, 47(4): 761-766. YANG H J, ZHAI Y F. Few-shot image classification algorithm base on D-ResNeXt backbone network[J]. Journal of Shanxi University (Natural Science Edition), 2024, 47(4): 761-766. (in Chinese)
[19] LIU J Y, YANG S, FANG Y M, et al. Structure-guided image inpainting using homography Transformation[J]. IEEE Transactions on Multimedia, 2018, 20(12): 3252-3265.
[20] YANG S, WANG Z Y, LIU J Y, et al. Deep plastic surgery: robust and controllable image editing with human-drawn sketches[C]//Proceedings of Europer Conference on Computer Vision. Berlin, Germany: Springer, 2020: 601-617.
[21] LIN J Y, WANG Y G. TSFormer: tracking structure transformer for image inpainting[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2024, 20(12): 1-23.
[22] 朱凯, 李理, 张彤, 等. 基于Transformer的多阶段运动模糊图像修复网络[J]. 计算机工程, 2024, 50(9): 276-285. ZHU K, LI L, ZHANG T, et al. Multi-stage motion blur image restoration network based on Transformer[J]. Computer Engineering, 2024, 50(9): 276-285. (in Chinese)
[23] DONG Q L, CAO C J, FU Y W. Incremental transformer structure enhanced image inpainting with masking positional encoding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE Press, 2022: 11348-11358.
[24] HEUSEL M, RAMSAUER H, UNTERTHINER T, et al. GANs trained by a two time-scale update rule converge to a local Nash equilibrium[EB/OL].[2025-05-15]. https://arxiv.org/pdf/1706.08500.
[25] WAN Z Y, ZHANG J B, CHEN D D, et al. High-fidelity pluralistic image completion with transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2021: 4672-4681.

选择文件类型/文献管理软件名称

选择包含的内容