Super-Resolution Reconstruction of Arbitrary Scale Images Based on Multi-Resolution Feature Fusion

doi:10.19678/j.issn.1000-3428.0065689

Abstract

Abstract:

Traditional deep learning image super-resolution reconstruction network only extracts features at a fixed resolution and cannot integrate advanced semantic information. The challenges include difficulties integrating advanced semantic information, reconstructing images with specific scale factors, limited generalization capability, and managing an excessive number of network parameters. An arbitrary scale image super-resolution reconstruction algorithm based on multi-resolution feature fusion is proposed, termed as MFSR. In the phase of multi-resolution feature fusion encoding, a multi-resolution feature extraction module is designed to extract different resolution features. A dual attention module is constructed to enhance the network feature extraction ability. The information-rich fused feature map is obtained by fully interacting with different resolution features. In the phase of image reconstruction, the fused feature map is decoded by a multi-layer perception machine to realize a super-resolution image at any scale. The experimental results indicate that tests were conducted on the Set5 data set with scaling factors of 2, 3, 4, 6, 8, and the Peak Signal-to-Noise Ratios (PSNR) of the proposed algorithm were 38.62, 34.70, 32.41, 28.96, and 26.62 dB, respectively. The model parameters correspond to 0.72×10⁶, which significantly reduce the number of parameters, maintain the reconstruction quality, and realize super-resolution image reconstruction at any scale. Furthermore, the model can realize better performance than mainstream algorithms, such as SRCNN, VDSR, and EDSR.

Key words: multi-resolution feature fusion, Super-Resolution Reconstruction(SRR), arbitrary scale, double attention, feature interaction

摘要：

传统深度学习的图像超分辨率重建网络仅在固定分辨率上提取特征，存在无法综合高级语义信息、只能以特定尺度因子重建图像、泛化能力较弱、网络参数量较大等问题。提出一种基于多分辨率特征融合的任意尺度图像超分辨率重建算法MFSR。在多分辨率特征融合编码阶段设计多分辨率特征提取模块以提取不同分辨率特征, 通过构建双重注意力模块增强网络特征提取能力，使不同分辨率特征之间进行充分交互, 以获取信息丰富的融合特征图。在图像重建阶段利用多层感知机对融合特征图进行解码，实现任意尺度的图像超分辨率重建。实验结果表明，在Set5数据集上分别以尺度因子2、3、4、6、8进行测试，所提算法的峰值信噪比分别为38.62、34.70、32.41、28.96、26.62 dB，模型参数量为0.72×10⁶，在大幅减少参数量的同时能保持重建质量，可以实现任意尺度的图像超分辨率重建，性能优于SRCNN、VDSR、EDSR等主流算法。

关键词: 多分辨率特征融合, 超分辨率重建, 任意尺度, 双重注意力, 特征交互

Wenzhuo FAN, Tao WU, Junping XU, Qingqing LI, Jianlin ZHANG, Meihui LI, Yuxing WEI. Super-Resolution Reconstruction of Arbitrary Scale Images Based on Multi-Resolution Feature Fusion[J]. Computer Engineering, 2023, 49(9): 217-225.

范文卓, 吴涛, 许俊平, 李庆庆, 张建林, 李美惠, 魏宇星. 基于多分辨率特征融合的任意尺度图像超分辨率重建[J]. 计算机工程, 2023, 49(9): 217-225.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0065689

http://www.ecice06.com/EN/Y2023/V49/I9/217

Figures/Tables 10

Fig.1 SE module

Fig.2 Structure of MFSR algorithm

Fig.3 Multi-resolution feature fusion module

Fig.4 Double attention block

Fig.5 PSNR values and parameters comparison among different algorithms

Fig.6 Comparison of the reconstruction results of different algorithms with a scaling factor of 2

Fig.7 Comparison of the reconstruction results of different algorithms with a scaling factor of 3

Fig.8 Comparison of the reconstruction results of different algorithms with a scaling factor of 4

References 27

1	MOSTOFA M, FERDOUS S N, RIGGAN B S, et al. Joint-SRVDNet: joint super resolution and vehicle detection network. IEEE Access, 2020, 8, 82306- 82319. doi: 10.1109/ACCESS.2020.2990870
2	NOH J, BAE W, LEE W, et al. Better to follow, follow to be better: towards precise supervision of feature super-resolution for small object detection[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2020: 9724-9733.
3	LI Z, ZHANG Y. Hyperspectral anomaly detection via image super-resolution processing and spatial correlation. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59 (3): 2307- 2320. doi: 10.1109/TGRS.2020.3005924
4	YOON Y, JEON H G, YOO D, et al. Learning a deep convolutional network for light-field image super-resolution[C]//Proceedings of IEEE International Conference on Computer Vision Workshop. Washington D. C., USA: IEEE Press, 2016: 57-65.
5	DONG C, LOY C C, TANG X O. Accelerating the super-resolution convolutional neural network[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 391-407.
6	SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 1874-1883.
7	KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 1646-1654.
8	LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington D. C., USA: IEEE Press, 2017: 1132-1140.
9	ZHANG Y L, TIAN Y P, KONG Y, et al. Residual dense network for image super-resolution[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 2472-2481.
10	斯捷, 肖雄, 李泾, 等. 基于生成对抗网络的多幅离焦图像超分辨率重建算法. 计算机工程, 2021, 47 (9): 266- 273. URL
	SI J, XIAO X, LI J, et al. Super-resolution reconstruction algorithm with multi-frame defocused images based on generative adversarial network. Computer Engineering, 2021, 47 (9): 266- 273. URL
11	姜玉宁, 李劲华, 赵俊莉. 基于生成式对抗网络的图像超分辨率重建算法. 计算机工程, 2021, 47 (3): 249- 255. URL
	JIANG Y N, LI J H, ZHAO J L. Image super-resolution reconstruction algorithm based on generative adversarial networks. Computer Engineering, 2021, 47 (3): 249- 255. URL
12	CHEN Y B, LIU S F, WANG X L. Learning continuous image representation with local implicit image function[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 8624-8634.
13	KONG X T, ZHAO H Y, QIAO Y, et al. ClassSR: a general framework to accelerate super-resolution networks by data characteristic[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 12011-12020.
14	柳聪, 屈丹, 司念文, 等. 基于深度可分离卷积的轻量级图像超分辨率重建. 计算机工程, 2022, 48 (6): 228- 234. URL
	LIU C, QU D, SI N W, et al. Lightweight image super-resolution reconstruction based on depthwise separable convolution. Computer Engineering, 2022, 48 (6): 228- 234. URL
15	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7132-7141.
16	DUPONT E, TEH Y W, DOUCET A. Generative models as distributions of functions[EB/OL]. [2022-08-01]. https://arxiv.org/abs/2102.04776.
17	SITZMANN V, MARTEL J N P, BERGMAN A W, et al. Implicit neural representations with periodic activation functions[EB/OL]. [2022-08-01]. https://arxiv.org/abs/2006.09661.
18	HU X C, MU H Y, ZHANG X Y, et al. Meta-SR: a magnification-arbitrary network for super-resolution[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 1575-1584.
19	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Berlin, Germany: Springer, 2018: 3-19.
20	FU J, LIU J, TIAN H J, et al. Dual attention network for scene segmentation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 3141-3149.
21	AGUSTSSON E, TIMOFTE R. NTIRE 2017 challenge on single image super-resolution: dataset and study[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington D. C., USA: IEEE Press, 2017: 1122-1131.
22	RADU T, AGUSTSSON F, VAN GOOL L, et al. Ntire 2017 challenge on single image super-resolution: methods and results[EB-OL]. [2022-08-01]. https://www.zhangqiaokeyan.com/academic-conference-foreign_meeting_thesis/0705011255254.html.
23	BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[EB/OL]. [2022-08-01]. https://www.mendeley.com/catalogue/c15e40be-7449-38e6-827d-5e11c62f3ce4/.
24	ZEYDE R, ELAD M, PROTTER M. On single image scale-up using sparse-representations[M]// BOISSONNAT J D, CHENIN P, COHEN A, et al. Curves and surfaces. Berlin, Germany: Springer, 2012: 711-730.
25	MARTIN D, FOWLKES C, TAL D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings of the 8th IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2002: 416-423.
26	DAI J F, QI H Z, XIONG Y W, et al. Deformable convolutional networks[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 764-773.
27	ZHANG Y L, LI K P, LI K, et al. Image super-resolution using very deep residual channel attention networks[EB/OL]. [2022-08-01]. https://www.semanticscholar.org/paper/Image-Super-Resolution-Using-Very-Deep-Residual-Zhang-Li/9775f8964a2eea1c9e35a02b1b906487396ea1f5.

[1]	SHEN Xueli, HAN Qianwen. Field-Aware Click-Through Rate Prediction Model Based on Attention Mechanism [J]. Computer Engineering, 2023, 49(3): 80-86,94.
[2]	ZHAO Yue, WU Zhihao, ZHAO Yiji. CTR Prediction Method Based on Feature and Domain Perception [J]. Computer Engineering, 2022, 48(3): 60-68.
[3]	LIAO Tao, HUANG Rongmei, ZHANG Shunxiang, DUAN Songsong. Nested Named Entity Recognition Based on Interactive Feature Fusion [J]. Computer Engineering, 2022, 48(12): 119-126,133.
[4]	NI Zhiwen, MA Xiaohu, SUN Xiao, BIAN Lina. Deep Fusion Model Combining Explicit and Implicit Feature Interactions [J]. Computer Engineering, 2020, 46(3): 87-92,98.
[5]	PENG Zhan,LIANG Gen,ZHOU Bing. Z Specification and Verification of Feature Interaction in Telecommunication Service System [J]. Computer Engineering, 2016, 42(8): 19-23.
[6]	ZHANG Rui; LIU Xiao-xia. Feature Interaction Filtering Method Based on URN [J]. Computer Engineering, 2009, 35(21): 45-47.
[7]	LI Miao; WU Shizhong. Research on Taxonomy of the Software Vulnerabilities Origins [J]. Computer Engineering, 2006, 32(20): 163-165.
[8]	WANG Jiahui;LU Yiqin;LI Wenle. Telecommunication Services Modeling Based on CPN Tools and Its Application [J]. Computer Engineering, 2006, 32(14): 221-223.

Please choose a citation manager

Content to export