Multi-Stage Motion Blur Image Restoration Network Based on Transformer

doi:10.19678/j.issn.1000-3428.0068246

Abstract

Abstract:

Motion blur is a common cause of image degradation that limits image readability and subsequent processing. A multi-stage deblurring network based on the Transformer is proposed to address the limited receptive field of convolutional networks and information loss in conventional multi-stage networks. The network adopts a multi-stage encoder-decoder structure with skip connections within and between stages to enhance information propagation. First, an efficient Transformer module is used to process the global and local information of the image using channel attention and depthwise convolution. Second, a multi-branch feedforward network with multiple parallel branches is introduced to extract and fuse features at different scales and levels. Finally, superior image restoration results are achieved through multi-stage residual learning. Experimental results show that the proposed method achieves a Peak Signal-to-Noise Ratio (PSNR) of 32.23 dB and Structural Similarity Index Measure (SSIM) of 0.955 on the GoPro dataset, and a PSNR of 30.15 dB and SSIM of 0.930 on the HIDE dataset, demonstrating a performance superior to DeepDeblur, DeblurGAN-V2, and other models.

Key words: deep learning, Transformer model, attention mechanism, image restoration, multi-scale network

摘要：

运动模糊是导致图像退化的常见原因, 其限制了图像的可读性和后续处理效果。针对卷积网络感受野有限以及常规多阶段网络中信息丢失的问题, 提出一种基于Transformer的多阶段去模糊网络。网络采用多阶段编码器-解码器结构, 在单个阶段内和多个阶段间采用跳跃连接来增强信息的传递。首先, 高效Transformer模块采用通道注意力和深度卷积来处理图像的全局和局部信息; 其次, 多分支结构的前馈传播网络通过引入多个并行的分支, 实现了不同尺度和不同层次的特征提取和融合; 最后, 通过多阶段的残差处理实现更优的图像恢复结果。实验结果显示, 在GoPro数据集上该网络的峰值信噪比(PSNR)达到32.23 dB, 结构相似性指数(SSIM)达到0.955, 在HIDE数据集上PSNR和SSIM分别达到30.15 dB和0.930, 优于DeepDeblur、DeblurGAN-V2等模型。

关键词: 深度学习, Transformer模型, 注意力机制, 图像修复, 多尺度网络

ZHU Kai, LI Li, ZHANG Tong, JIANG Sheng, BIE Yiming. Multi-Stage Motion Blur Image Restoration Network Based on Transformer[J]. Computer Engineering, 2024, 50(9): 276-285.

朱凯, 李理, 张彤, 江晟, 别一鸣. 基于Transformer的多阶段运动模糊图像修复网络[J]. 计算机工程, 2024, 50(9): 276-285.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0068246

https://www.ecice06.com/EN/Y2024/V50/I9/276

Figures/Tables 8

Fig.1 Different multi-stage deblurring model structures

Fig.2 SE module structure

Fig.3 Single-branch and multi-branch structure

Fig.4 Multi-stage deblurring network structure

Fig.5 Comparison of deblurring results of different models

References 46

1	KALMAN R E. A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 1960, 82(1): 35- 45. doi: 10.1115/1.3662552
2	LUCY L B. An iterative technique for the rectification of observed distributions. The Astronomical Journal, 1974, 79, 745. doi: 10.1086/111605
3	RICHARDSON W H. Bayesian-based iterative method of image restoration. Journal of the Optical Society of America, 1972, 62(1): 55. doi: 10.1364/JOSA.62.000055
4	KRISHNAN D, FERGUS R. Fast image deconvolution using hyper-Laplacian priors[EB/OL]. [2023-07-05]. https://proceedings.neurips.cc/paper/2009/file/3dd48ab31d016ffcbf3314df2b3cb9ce-Paper.pdf.
5	HRADIŠ M, KOTERA J, ZEM$ \stackrel{ˇ}{C} $ÍK P, et al. Convolutional neural networks for direct text deblurring[EB/OL]. [2023-07-05]. https://www.semanticscholar.org/paper/Convolutional-Neural-Networks-for-Direct-Text-Hradi%C5%A1-Kotera/423584105e5f6adc5981c87f2af8fc5ff2bb9064.
6	SCHULER C J, HIRSCH M, HARMELING S, et al. Learning to deblur. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(7): 1439- 1451. doi: 10.1109/TPAMI.2015.2481418
7	XU X, PAN J, ZHANG Y J, et al. Motion blur kernel estimation via deep learning. IEEE Transactions on Image Processing, 2018, 27(1): 194- 205. doi: 10.1109/TIP.2017.2753658
8	KUPYN O, BUDZAN V, MYKHAILYCH M, et al. DeblurGAN: blind motion deblurring using conditional adversarial networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 8183-8192.
9	ZHANG H G, DAI Y C, LI H D, et al. Deep stacked hierarchical multi-patch network for image deblurring[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 5971-5979.
10	PUROHIT K, RAJAGOPALAN A N. Motion deblurring with an adaptive network[EB/OL]. [2023-07-05]. http://arxiv.org/abs/1903.11394v4.
11	NAH S, KIM T H, LEE K M. Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 257-265.
12	DONG J X, PAN J S, YANG Z B, et al. Multi-scale residual low-pass filter network for image deblurring[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2023: 12345-12354.
13	LIU P J, ZHANG H Z, ZHANG K, et al. Multi-level wavelet-CNN for image restoration[EB/OL]. [2023-07-05]. https://arxiv.org/pdf/1805.07071.
14	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[EB/OL]. [2023-07-05]. https://arxiv.org/abs/2010.11929.
15	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL]. [2023-07-05]. https://arxiv.org/abs/1706.03762.
16	WANG Z D, CUN X D, BAO J M, et al. Uformer: a general U-shaped Transformer for image restoration[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 17662-17672.
17	LIU Z, LIN Y T, CAO Y, et al. Swin Transformer: hierarchical vision Transformer using shifted windows[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 9992-10002.
18	ZAMIR S W, ARORA A, KHAN S, et al. Restormer: efficient Transformer for high-resolution image restoration[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 5718-5729.
19	ZHAO Q, YANG H, ZHOU D M, et al. Rethinking image deblurring via CNN-Transformer multiscale hybrid architecture. IEEE Transactions on Instrumentation Measurement, 2023, 72, 3230482.
20	KONG L S, DONG J X, GE J J, et al. Efficient frequency domain-based transformers for high-quality image deblurring[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2023: 5886-5895.
21	CUI Y N, TAO Y, REN W Q, et al. Dual-domain attention for image deblurring. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(1): 479- 487. doi: 10.1609/aaai.v37i1.25122
22	KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 1646-1654.
23	TAO X, GAO H Y, SHEN X Y, et al. Scale-recurrent network for deep image deblurring[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 8174-8182.
24	CHO S J, JI S W, HONG J P, et al. Rethinking coarse-to-fine approach in single image deblurring[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 4621-4630.
25	王向军, 欧阳文森. 多尺度循环注意力网络运动模糊图像复原方法. 红外与激光工程, 2022, 51(6): 460- 468. URL
	WANG X J, OUYANG W S. Multi-scale recurrent attention network for image motion deblurring. Infrared and Laser Engineering, 2022, 51(6): 460- 468. URL
26	ZHANG Y, LI Q, QI M, et al. Multi-scale frequency separation network for image deblurring[EB/OL]. [2023-07-05]. https://arxiv.org/abs/2206.00798.
27	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[EB/OL]. [2023-07-05]. https://arxiv.org/abs/1807.06521.
28	HASSANI A, WALTON S, LI J C, et al. Neighborhood attention Transformer[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2023: 6185-6194.
29	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[EB/OL]. [2023-07-05]. https://arxiv.org/abs/1505.04597.
30	GAO H Y, TAO X, SHEN X Y, et al. Dynamic scene deblurring with parameter selective sharing and nested skip connections[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 3843-3851.
31	HU J, SHEN L, SUN G. Squeeze-and-Excitation networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7132-7141.
32	SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 2818-2826.
33	TSAI F J, PENG Y T, LIN Y Y, et al. Stripformer: strip Transformer for fast image deblurring[EB/OL]. [2023-07-05]. https://arxiv.org/abs/2204.04627.
34	SHI W Z, CABALLERO J, HUSZAR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 1874-1883.
35	赵敬伟, 林珊玲, 梅婷, 等. 基于YOLACT与Transformer相结合的实例分割算法研究. 半导体光电, 2023, 44(1): 134- 140. URL
	ZHAO J W, LIN S L, MEI T, et al. Research on instance segmentation algorithm based on YOLACT and Transformer. Semiconductor Optoelectronics, 2023, 44(1): 134- 140. URL
36	CHEN C F, PANDA R, FAN Q F. RegionViT: regional-to-local attention for Vision Transformers[EB/OL]. [2023-07-05]. http://arxiv.org/abs/2106.02689v3.
37	ZHANG H, WU C R, ZHANG Z Y, et al. ResNeSt: split-attention networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 2735-2745.
38	LIANG J Y, CAO J Z, SUN G L, et al. SwinIR: image restoration using Swin Transformer[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 1833-1844.
39	SHEN Z Y, WANG W G, LU X K, et al. Human-aware motion deblurring[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 5572-5581.
40	LOSHCHILOV I, HUTTER F. SGDR: stochastic gradient descent with warm restarts[EB/OL]. [2023-07-05]. http://arxiv.org/abs/1608.03983v5.
41	KUPYN O, MARTYNIUK T, WU J R, et al. DeblurGAN-V2: deblurring (orders-of-magnitude) faster and better[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 8877-8886.
42	PARK D, KANG D U, KIM J, et al. Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training[EB/OL]. [2023-07-05]. https://arxiv.org/abs/1911.07410.
43	WAN S D, TANG S, XIE X Z, et al. Deep convolutional-neural-network-based channel attention for single image dynamic scene blind deblurring. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(8): 2994- 3009. doi: 10.1109/TCSVT.2020.3035664
44	PUROHIT K, SUIN M, RAJAGOPALAN A N, et al. Spatially-adaptive image restoration using distortion-guided networks[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 2289-2299.
45	WANG J B, WANG Z Q, YANG A P. Iterative dual CNNs for image deblurring. Mathematics, 2022, 10(20): 3891. doi: 10.3390/math10203891
46	李现国, 李滨. 基于Transformer和多尺度CNN的图像去模糊. 计算机工程, 2023, 49(9): 226-233, 245. URL
	LI X G, LI B. Image deblurring based on Transformer and multi-scale CNN. Computer Engineering, 2023, 49(9): 226-233, 245. URL

[1]	LI Junjun, DONG Jiangang, LI Kun. Research on Kubernetes-based Cluster Energy-Saving Strategy [J]. Computer Engineering, 2024, 50(9): 82-91.
[2]	LIN Chang, GUO Wei, REN Zhecong, JIN Haibo. Unification Algorithm for Object Tracking and Segmentation Based on Transformer [J]. Computer Engineering, 2024, 50(9): 130-141.
[3]	LI Zelin, LÜ Zhaofeng, CHEN Fuqiang, LI Ke. Entity Alignment Model Based on Multi-Hop Information Fusion [J]. Computer Engineering, 2024, 50(9): 142-152.
[4]	WANG Ruying, MA Jiajun, DONG Jianqiang, LIU Wanlong, ZHANG Haitao, YIN Kai, ZHAO Bochao. Industrial Load Forecasting Method Based on MTS-BiGRU-DMHSA [J]. Computer Engineering, 2024, 50(9): 169-178.
[5]	ZHANG Tianpeng, HAN Jing, LÜ Xueqiang. Super-Resolution-Aided Small-Target Detection Based on Multi-Task Learning [J]. Computer Engineering, 2024, 50(9): 304-312.
[6]	GUO Min, ZHANG Xihan, LI Yang. Integrated Attentional Teacher Mutual Consistency Semi-Supervised Medical Image Segmentation [J]. Computer Engineering, 2024, 50(9): 313-323.
[7]	GAO Yubao, WEN Zhicheng. Dual Decoder Image Denoising Method Based on Attention Mechanism [J]. Computer Engineering, 2024, 50(9): 324-332.
[8]	ZENG Yuqi, LIU Bo, ZHONG Baichang, ZHONG Jin. Student Classroom Behavior Detection Algorithm Based on Improved YOLOv8 in Smart Education [J]. Computer Engineering, 2024, 50(9): 344-355.
[9]	WANG Yanguo, LÜ Pengyuan, LAN Jinjiang, LIU Mingzhe, QIN Guanjun, ZHANG Shuohua, ZHOU Yu. Wind Turbine Fault Classification Method Based on Adversarial Training and Transformer [J]. Computer Engineering, 2024, 50(9): 377-384.
[10]	WEI Wei, DING Xiangxiang, GUO Mengxing, YANG Zhao, LIU Hui. Review of Text Similarity Calculation Methods [J]. Computer Engineering, 2024, 50(9): 18-32.
[11]	Xuebing BAI, Jin CHE, Jinman WU, Yumin CHEN. Image Captioning Method Based on Transformer Visual Features Fusion [J]. Computer Engineering, 2024, 50(8): 229-238.
[12]	Hong ZHAO, Xiao WANG. Study on Lesion Segmentation of Melanoma Images Based on Swin-Transformer [J]. Computer Engineering, 2024, 50(8): 249-258.
[13]	Suzhe WANG, Xueying ZHANG, Xiaoyu CHEN, Fenglian LI, Zeling WU. EEG Enhancement Algorithm Based on Combination of Effective Attention and GAN [J]. Computer Engineering, 2024, 50(8): 336-344.
[14]	Wei GAO, Shuailong LI, Lin MAO, Lei WANG, Yingying LI, Lin HAN. One Acceleration Strategy for Operator Generation Based on TVM [J]. Computer Engineering, 2024, 50(8): 353-362.
[15]	Yu WANG, Qi QI, Chun WANG, Cai XU. High-Precision Fault Diagnosis Method for Energy Storage Inverter Signals [J]. Computer Engineering, 2024, 50(8): 389-396.

Please choose a citation manager

Content to export