基于多分辨率特征融合的任意尺度图像超分辨率重建

doi:10.19678/j.issn.1000-3428.0065689

计算机工程 ›› 2023, Vol. 49 ›› Issue (9): 217-225. doi: 10.19678/j.issn.1000-3428.0065689

基于多分辨率特征融合的任意尺度图像超分辨率重建

范文卓¹^,², 吴涛¹^,², 许俊平²^,*, 李庆庆², 张建林², 李美惠², 魏宇星²

1. 中国科学院大学电子电气与通信工程学院, 北京 101408
2. 中国科学院光电技术研究所光束控制重点实验室, 成都 610207

收稿日期:2022-09-05 出版日期:2023-09-15 发布日期:2023-09-14
通讯作者: 许俊平
作者简介:
范文卓（1995—），男，硕士研究生，主研方向为图像超分辨率重建
吴涛，硕士研究生
李庆庆，助理研究员、博士
张建林，研究员、博士
李美惠，博士
魏宇星，副研究员
基金资助:
国家自然科学基金青年基金(62101529)

Super-Resolution Reconstruction of Arbitrary Scale Images Based on Multi-Resolution Feature Fusion

Wenzhuo FAN¹^,², Tao WU¹^,², Junping XU²^,*, Qingqing LI², Jianlin ZHANG², Meihui LI², Yuxing WEI²

1. School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 101408, China
2. Key Laboratory of Beam Control, Institute of Optics and Electronics, Chinese Academy of Sciences, Chengdu 610207, China

Received:2022-09-05 Online:2023-09-15 Published:2023-09-14
Contact: Junping XU

摘要/Abstract

摘要：

传统深度学习的图像超分辨率重建网络仅在固定分辨率上提取特征，存在无法综合高级语义信息、只能以特定尺度因子重建图像、泛化能力较弱、网络参数量较大等问题。提出一种基于多分辨率特征融合的任意尺度图像超分辨率重建算法MFSR。在多分辨率特征融合编码阶段设计多分辨率特征提取模块以提取不同分辨率特征, 通过构建双重注意力模块增强网络特征提取能力，使不同分辨率特征之间进行充分交互, 以获取信息丰富的融合特征图。在图像重建阶段利用多层感知机对融合特征图进行解码，实现任意尺度的图像超分辨率重建。实验结果表明，在Set5数据集上分别以尺度因子2、3、4、6、8进行测试，所提算法的峰值信噪比分别为38.62、34.70、32.41、28.96、26.62 dB，模型参数量为0.72×10⁶，在大幅减少参数量的同时能保持重建质量，可以实现任意尺度的图像超分辨率重建，性能优于SRCNN、VDSR、EDSR等主流算法。

关键词: 多分辨率特征融合, 超分辨率重建, 任意尺度, 双重注意力, 特征交互

Abstract:

Traditional deep learning image super-resolution reconstruction network only extracts features at a fixed resolution and cannot integrate advanced semantic information. The challenges include difficulties integrating advanced semantic information, reconstructing images with specific scale factors, limited generalization capability, and managing an excessive number of network parameters. An arbitrary scale image super-resolution reconstruction algorithm based on multi-resolution feature fusion is proposed, termed as MFSR. In the phase of multi-resolution feature fusion encoding, a multi-resolution feature extraction module is designed to extract different resolution features. A dual attention module is constructed to enhance the network feature extraction ability. The information-rich fused feature map is obtained by fully interacting with different resolution features. In the phase of image reconstruction, the fused feature map is decoded by a multi-layer perception machine to realize a super-resolution image at any scale. The experimental results indicate that tests were conducted on the Set5 data set with scaling factors of 2, 3, 4, 6, 8, and the Peak Signal-to-Noise Ratios (PSNR) of the proposed algorithm were 38.62, 34.70, 32.41, 28.96, and 26.62 dB, respectively. The model parameters correspond to 0.72×10⁶, which significantly reduce the number of parameters, maintain the reconstruction quality, and realize super-resolution image reconstruction at any scale. Furthermore, the model can realize better performance than mainstream algorithms, such as SRCNN, VDSR, and EDSR.

Key words: multi-resolution feature fusion, Super-Resolution Reconstruction(SRR), arbitrary scale, double attention, feature interaction

范文卓, 吴涛, 许俊平, 李庆庆, 张建林, 李美惠, 魏宇星. 基于多分辨率特征融合的任意尺度图像超分辨率重建[J]. 计算机工程, 2023, 49(9): 217-225.

Wenzhuo FAN, Tao WU, Junping XU, Qingqing LI, Jianlin ZHANG, Meihui LI, Yuxing WEI. Super-Resolution Reconstruction of Arbitrary Scale Images Based on Multi-Resolution Feature Fusion[J]. Computer Engineering, 2023, 49(9): 217-225.

https://www.ecice06.com/CN/Y2023/V49/I9/217

图/表 10

图1 SE模块

Fig.1 SE module

图2 MFSR算法结构

Fig.2 Structure of MFSR algorithm

图3 多分辨率特征融合模块

Fig.3 Multi-resolution feature fusion module

图4 双重注意力模块

Fig.4 Double attention block

图5 不同算法的PSNR值及参数量对比

Fig.5 PSNR values and parameters comparison among different algorithms

图6 尺度因子为2时各算法的重建结果对比

Fig.6 Comparison of the reconstruction results of different algorithms with a scaling factor of 2

图7 尺度因子为3时各算法的重建结果对比

Fig.7 Comparison of the reconstruction results of different algorithms with a scaling factor of 3

图8 尺度因子为4时各算法的重建结果对比

Fig.8 Comparison of the reconstruction results of different algorithms with a scaling factor of 4

参考文献 27

1	MOSTOFA M, FERDOUS S N, RIGGAN B S, et al. Joint-SRVDNet: joint super resolution and vehicle detection network. IEEE Access, 2020, 8, 82306- 82319. doi: 10.1109/ACCESS.2020.2990870
2	NOH J, BAE W, LEE W, et al. Better to follow, follow to be better: towards precise supervision of feature super-resolution for small object detection[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2020: 9724-9733.
3	LI Z, ZHANG Y. Hyperspectral anomaly detection via image super-resolution processing and spatial correlation. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59 (3): 2307- 2320. doi: 10.1109/TGRS.2020.3005924
4	YOON Y, JEON H G, YOO D, et al. Learning a deep convolutional network for light-field image super-resolution[C]//Proceedings of IEEE International Conference on Computer Vision Workshop. Washington D. C., USA: IEEE Press, 2016: 57-65.
5	DONG C, LOY C C, TANG X O. Accelerating the super-resolution convolutional neural network[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 391-407.
6	SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 1874-1883.
7	KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 1646-1654.
8	LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington D. C., USA: IEEE Press, 2017: 1132-1140.
9	ZHANG Y L, TIAN Y P, KONG Y, et al. Residual dense network for image super-resolution[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 2472-2481.
10	斯捷, 肖雄, 李泾, 等. 基于生成对抗网络的多幅离焦图像超分辨率重建算法. 计算机工程, 2021, 47 (9): 266- 273. URL
	SI J, XIAO X, LI J, et al. Super-resolution reconstruction algorithm with multi-frame defocused images based on generative adversarial network. Computer Engineering, 2021, 47 (9): 266- 273. URL
11	姜玉宁, 李劲华, 赵俊莉. 基于生成式对抗网络的图像超分辨率重建算法. 计算机工程, 2021, 47 (3): 249- 255. URL
	JIANG Y N, LI J H, ZHAO J L. Image super-resolution reconstruction algorithm based on generative adversarial networks. Computer Engineering, 2021, 47 (3): 249- 255. URL
12	CHEN Y B, LIU S F, WANG X L. Learning continuous image representation with local implicit image function[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 8624-8634.
13	KONG X T, ZHAO H Y, QIAO Y, et al. ClassSR: a general framework to accelerate super-resolution networks by data characteristic[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 12011-12020.
14	柳聪, 屈丹, 司念文, 等. 基于深度可分离卷积的轻量级图像超分辨率重建. 计算机工程, 2022, 48 (6): 228- 234. URL
	LIU C, QU D, SI N W, et al. Lightweight image super-resolution reconstruction based on depthwise separable convolution. Computer Engineering, 2022, 48 (6): 228- 234. URL
15	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7132-7141.
16	DUPONT E, TEH Y W, DOUCET A. Generative models as distributions of functions[EB/OL]. [2022-08-01]. https://arxiv.org/abs/2102.04776.
17	SITZMANN V, MARTEL J N P, BERGMAN A W, et al. Implicit neural representations with periodic activation functions[EB/OL]. [2022-08-01]. https://arxiv.org/abs/2006.09661.
18	HU X C, MU H Y, ZHANG X Y, et al. Meta-SR: a magnification-arbitrary network for super-resolution[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 1575-1584.
19	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Berlin, Germany: Springer, 2018: 3-19.
20	FU J, LIU J, TIAN H J, et al. Dual attention network for scene segmentation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 3141-3149.
21	AGUSTSSON E, TIMOFTE R. NTIRE 2017 challenge on single image super-resolution: dataset and study[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington D. C., USA: IEEE Press, 2017: 1122-1131.
22	RADU T, AGUSTSSON F, VAN GOOL L, et al. Ntire 2017 challenge on single image super-resolution: methods and results[EB-OL]. [2022-08-01]. https://www.zhangqiaokeyan.com/academic-conference-foreign_meeting_thesis/0705011255254.html.
23	BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[EB/OL]. [2022-08-01]. https://www.mendeley.com/catalogue/c15e40be-7449-38e6-827d-5e11c62f3ce4/.
24	ZEYDE R, ELAD M, PROTTER M. On single image scale-up using sparse-representations[M]// BOISSONNAT J D, CHENIN P, COHEN A, et al. Curves and surfaces. Berlin, Germany: Springer, 2012: 711-730.
25	MARTIN D, FOWLKES C, TAL D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings of the 8th IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2002: 416-423.
26	DAI J F, QI H Z, XIONG Y W, et al. Deformable convolutional networks[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 764-773.
27	ZHANG Y L, LI K P, LI K, et al. Image super-resolution using very deep residual channel attention networks[EB/OL]. [2022-08-01]. https://www.semanticscholar.org/paper/Image-Super-Resolution-Using-Very-Deep-Residual-Zhang-Li/9775f8964a2eea1c9e35a02b1b906487396ea1f5.

[1]	杨郅树, 梁佳楠, 曹永军, 钟震宇, 何永伦. 基于局部分离与多尺度融合的图像超分辨率重建[J]. 计算机工程, 2024, 50(7): 314-323.
[2]	宋志娜, 李莎, 杨建明, 徐川. 基于特征与区域定位增强的遥感舰船目标检测[J]. 计算机工程, 2023, 49(8): 257-264.
[3]	丁子轩, 俞雷, 张娟, 李想, 王新宇. 基于深度残差自适应注意力网络的图像超分辨率重建[J]. 计算机工程, 2023, 49(5): 231-238.
[4]	李培育, 张雅丽. 基于改进SRGAN模型的人脸图像超分辨率重建[J]. 计算机工程, 2023, 49(4): 199-205.
[5]	沈学利, 韩倩雯. 基于注意力机制的场感知点击率预测模型[J]. 计算机工程, 2023, 49(3): 80-86,94.
[6]	黎浩民, 李光平. 基于稀疏神经网络的图像超分辨率重建算法[J]. 计算机工程, 2022, 48(7): 247-253.
[7]	柳聪, 屈丹, 司念文, 魏紫薇. 基于深度可分离卷积的轻量级图像超分辨率重建[J]. 计算机工程, 2022, 48(6): 228-234.
[8]	赵越, 武志昊, 赵苡积. 基于特征与域感知的点击率预估方法[J]. 计算机工程, 2022, 48(3): 60-68.
[9]	楼鑫杰, 李小薪, 刘志勇. 基于反馈机制的图像超分辨率重建算法[J]. 计算机工程, 2022, 48(2): 261-267.
[10]	廖涛, 黄荣梅, 张顺香, 段松松. 基于交互式特征融合的嵌套命名实体识别[J]. 计算机工程, 2022, 48(12): 119-126,133.
[11]	张法正, 杨娟, 汪荣贵, 薛丽霞. 基于动态自适应层叠网络的轻量化图像超分辨率重建[J]. 计算机工程, 2022, 48(12): 196-202.
[12]	曾涛, 薛峰, 杨添. 面向行人重识别的通道与空间双重注意力网络[J]. 计算机工程, 2022, 48(12): 281-287,295.
[13]	陈乔松, 蒲柳, 张羽, 孙开伟, 邓欣, 王进. 结合整体注意力与分形稠密特征的图像超分辨率重建[J]. 计算机工程, 2022, 48(11): 207-214,223.
[14]	姜红涛, 孙京, 谢成, 赖少川, 沈焕锋. 内外部梯度联合约束的图像超分辨率重建方法[J]. 计算机工程, 2022, 48(1): 220-227,235.
[15]	斯捷, 肖雄, 李泾, 马明勋, 毛玉星. 基于生成对抗网络的多幅离焦图像超分辨率重建算法[J]. 计算机工程, 2021, 47(9): 266-273.

选择文件类型/文献管理软件名称

选择包含的内容

基于多分辨率特征融合的任意尺度图像超分辨率重建

Super-Resolution Reconstruction of Arbitrary Scale Images Based on Multi-Resolution Feature Fusion

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 27

相关文章 15

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于多分辨率特征融合的任意尺度图像超分辨率重建

Super-Resolution Reconstruction of Arbitrary Scale Images Based on Multi-Resolution Feature Fusion

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 27

相关文章 15

编辑推荐

Metrics

本文评价