基于局部分离与多尺度融合的图像超分辨率重建

doi:10.19678/j.issn.1000-3428.0067857

摘要/Abstract

摘要：

目前基于深度学习的超分辨率重建网络存在卷积运算冗余、图像重建信息不完整、模型参数庞大等问题,限制了其在边缘设备上的适用性。针对上述问题,提出一种轻量级的局部分离与多尺度融合图像超分辨率重建网络,该网络利用局部卷积对图像进行特征提取,通过分离部分图像通道,在减少网络冗余计算的同时保持图像重建的质量。设计一种多尺度特征融合模块,在空间维度学习长依赖特征,并采用一个通道注意力增强组在空间维度捕获空间特征,减少图像重建信息的丢失,有效恢复图像的细节纹理。由于多尺度特征融合模块更多地是从全局角度进行特征提取融合,因此构建一种高效反残差模块补充网络的局部上下文信息提取能力。在Set 5、Set 14、B 100、Urban 100、Manga 109这5个基准数据集上的实验结果表明,当尺度因子为2、3、4倍时,该网络的参数量分别为373 000、382 000、394 000,FLOPs分别为84.0×10⁹、38.1×10⁹、22.1×10⁹。与VDSR、IMDN、RFDN、RLFN等网络相比,该网络在较少网络参数的情况下,能够保证图像重建效果。

关键词: 超分辨率重建, 轻量级网络, 局部卷积, 多尺度融合, 长依赖关系

Abstract:

Currently, deep-learning-based super-resolution reconstruction networks suffer from issues such as convolution operation redundancy, incomplete image reconstruction information, and large model parameters that limit their applicability to edge devices. To address these issues, this study proposes a lightweight image super-resolution reconstruction network based on partial separation and multiscale fusion. This network utilizes partial convolutions for feature extraction and separates partial image channels to reduce redundant computations while maintaining the quality of the image reconstruction. At the same time, a multiscale feature fusion module is designed to learn long-range dependency features and capture spatial features in the spatial dimension using a channel attention enhancement group. This reduces the loss of image reconstruction information and effectively restores the details and textures of the image. Finally, because the multiscale feature fusion block focuses on global feature extraction and fusion, an efficient inverted residual block is constructed to supplement the ability to extract local contextual information. The network is tested on five benchmark datasets: Set 5, Set 14, B 100, Urban 100, and Manga 109, with scale factors of 2, 3, and 4 times. The parameters of the network are 373 000, 382 0000, and 394 000, and the FLOPs are 84.0×10⁹, 38.1×10⁹, and 22.1 ×10⁹, respectively. Quantitative and qualitative experimental results show that compared with networks such as VDSR, IMDN, RFDN, and RLFN, the proposed network ensures image reconstruction quality with fewer network parameters.

Key words: super-resolution reconstruction, lightweight network, partial convolution, multiscale fusion, longrange dependencies

杨郅树, 梁佳楠, 曹永军, 钟震宇, 何永伦. 基于局部分离与多尺度融合的图像超分辨率重建[J]. 计算机工程, 2024, 50(7): 314-323.

Zhishu YANG, Jianan LIANG, Yongjun CAO, Zhenyu ZHONG, Yonglun HE. Based on Partial Separation and Multiscale Fusion[J]. Computer Engineering, 2024, 50(7): 314-323.

https://www.ecice06.com/CN/Y2024/V50/I7/314

图/表 13

图1 局部分离与多尺度融合网络架构

Fig.1 Partial separation and multiscale fusion network architecture

图2 高效反残差块结构

Fig.2 Structure of efficient inverse residual block

图3 多尺度特征融合块

Fig.3 Multiscale feature fusion block

图4 不同网络的性能和复杂度对比

Fig.4 Comparison of performance and complexity of different networks

图5 4倍SR在Urban 100数据集上的可视化比较1

Fig.5 Visual comparisons for 4 times SR on the Urban 100 dataset 1

图6 4倍SR在Urban 100数据集上的可视化比较2

Fig.6 Visual comparisons for 4 times SR on the Urban 100 dataset 2

图7 4倍SR在Urban 100数据集上的可视化比较3

Fig.7 Visual comparisons for 4 times SR on the Urban 100 dataset 3

图8 特征图可视化

Fig.8 Feature map visualization

图9 MFFB和EIRB的作用

Fig.9 Effect of MFFB and EIRB

图10 局部归因图和扩散指数

Fig.10 Local attribution maps and diffusion indices

参考文献 37

1	DONG C, LOY C C, HE K M, et al. Learning a deep convolutional network for image super-resolution[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 184-199.
2	KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C. , USA: IEEE Press, 2016: 1646-1654.
3	ZHANG Y L, LI K P, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of European Conference on Computer Vision. Berlin, German: Springer, 2018: 294-310.
4	LIANG J Y, CAO J Z, SUN G L, et al. SwinIR: image restoration using swin transformer[C]//Proceedings of IEEE/CVF International Conference on Computer Vision Workshops. Washington D. C. , USA: IEEE Press, 2021: 1833-1844.
5	陈豪, 夏振平, 程成, 等. 基于Transformer-CNN的轻量级图像超分辨率重建网络. 计算机应用, 2024, 44 (1): 292- 299. URL
	CHEN H, XIA Z P, CHENG C, et al. Lightweight image super resolution reconstruction network based on Transformer-CNN. Journal of Computer Applications, 2024, 44 (1): 292- 299. URL
6	赵小强, 程伟. 基于空间特征交叉融合的轻量级图像超分辨率重建. 兵工学报, 2024, 45 (4): 1273- 1284. URL
	ZHAO X Q, CHENG W. Lightweight image super-resolution reconstruction based on cross-fusion of spatial features. Acta Armamentarii, 2024, 45 (4): 1273- 1284. URL
7	吕鑫栋, 李娇, 邓真楠, 等. 基于改进Transformer的结构化图像超分辨网络. 浙江大学学报(工学版), 2023, 57 (5): 865- 874,910. URL
	LÜ X D, LI J, DENG Z N, et al. Structured image super-resolution network based on improved Transformer. Journal of Zhejiang University (Engineering Science), 2023, 57 (5): 865- 874,910. URL
8	CHU X X, ZHANG B, MA H L, et al. Fast, accurate and lightweight super-resolution with neural architecture search[C]//Proceedings of the 25th International Conference on Pattern Recogniti. Washington D. C. , USA: IEEE Press, 2021: 59-64.
9	WANG X T, DONG C, SHAN Y. RepSR: training efficient VGG-style super-resolution networks with structural re-parameterization and batch normalization[EB/OL]. [2023-05-22]. http://arxiv.org/abs/2205.05671v1.
10	LIU J, TANG J, WU G S. Residual feature distillation network for lightweight image super-resolution[C]//Proceedings of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 41-55.
11	范文卓, 吴涛, 许俊平, 等. 基于多分辨率特征融合的任意尺度图像超分辨率重建. 计算机工程, 2023, 49 (9): 217- 225. URL
	FAN W Z, WU T, XU J P, et al. Super-resolution reconstruction of arbitrary scale images based on multi-resolution feature fusion. Computer Engineering, 2023, 49 (9): 217- 225. URL
12	丁子轩, 俞雷, 张娟, 等. 基于深度残差自适应注意力网络的图像超分辨率重建. 计算机工程, 2023, 49 (5): 231- 238. doi: 10.19678/j.issn.1000-3428.0064243
	DING Z X, YU L, ZHANG J, et al. Image super-resolution reconstruction based on depth residual adaptive attention network. Computer Engineering, 2023, 49 (5): 231- 238. doi: 10.19678/j.issn.1000-3428.0064243
13	高丹丹, 周登文, 王婉君, 等. 特征频率分组融合的轻量级图像超分辨率重建. 计算机辅助设计与图形学学报, 2023, 35 (7): 1020- 1031. URL
	GAO D D, ZHOU D W, WANG W J. Lightweight super-resolution via grouping fusion of feature frequencies. Journal of Computer-Aided Design & Computer Graphics, 2023, 35 (7): 1020- 1031. URL
14	CHEN J R, KAO S H, HE H, et al. Run, don’t walk: chasing higher FLOPS for faster neural networks[EB/OL]. [2023-05-22]. http://arxiv.org/abs/2303.03667v3.
15	LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington D. C. , USA: IEEE Press, 2017: 1132-1140.
16	AHN N, KANG B, SOHN K A. Fast, accurate, and lightweight super-resolution with cascading residual network[C]//Proceedings of the 15th European Conference on Computer Vision. Berlin, German: Springer, 2018: 256-272.
17	ZHANG J L, ZHANG Y L, GU J J, et al. Accurate image restoration with attention retractable transformer[EB/OL]. [2023-05-22]. http://arxiv. org/abs/2210.01427v4.
18	DONG C, LOY C C, TANG X O. Accelerating the super-resolution convolutional neural network[C]// Proceedings of European Conference on Computer Vision. Berlin, German: Springer, 2016: 391-407.
19	SHI W Z, CABALLERO J, HUSZAR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C. , USA: IEEE Press, 2016: 1874-1883.
20	ZHAO H Y, KONG X T, HE J W, et al. Efficient image super-resolution using pixel attention[C]// Proceedings of the 16th European Conference on Computer Vision. Berlin, German: Springer, 2020: 56-72.
21	HUI Z, GAO X B, YANG Y C, et al. Lightweight image super-resolution with information multi-distillation network[C]//Proceedings of the 27th ACM International Conference on Multimedia. New York, USA: ACM Press, 2019: 2024-2032.
22	DU Z C, LIU D, LIU J, et al. Fast and memory-efficient network towards efficient image super-resolution[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Washington D. C. , USA: IEEE Press, 2022: 852-861.
23	SANDLER M, HOWARD A, ZHU M L, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C. , USA: IEEE Press, 2018: 4510-4520.
24	SUN L, DONG J X, TANG J H, et al. Spatially-adaptive feature modulation for efficient image super-resolution[EB/OL]. [2023-05-22]. http://arxiv.org/abs/2302.13800v1.
25	LI Y W, FAN Y C, XIANG X Y, et al. Efficient and explicit modelling of image hierarchies for image restoration[EB/OL]. [2023-05-22]. http://arxiv.org/abs/2303.00748v2.
26	GUO M H, LU C Z, LIU Z N, et al. Visual attention network[EB/OL]. [2023-05-22]. http://arxiv.org/abs/2202.09741v5.
27	AGUSTSSON E, TIMOFTE R. NTIRE 2017 challenge on single image super-resolution: dataset and study[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington D. C. , USA: IEEE Press, 2017: 1122-1131.
28	BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of British Machine Vision Conference. Surrey, UK: British Machine Vision Association, 2012: 325-333.
29	ZEYDE R, ELAD M, PROTTER M. On single image scale-up using sparse-representations[C]//Proceedings of the 7th International Conference on Curves and Surfaces. New York, USA: ACM Press, 2010: 711-730.
30	MARTIN D, FOWLKES C, TAL D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings of the 8th IEEE International Conference on Computer Vision. Washington D. C. , USA: IEEE Press, 2001: 416-423.
31	HUANG J B, SINGH A, AHUJA N. Single image super-resolution from transformed self-exemplars[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C. , USA: IEEE Press, 2015: 5197-5206.
32	MATSUI Y, ITO K, ARAMAKI Y, et al. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications, 2017, 76 (20): 21811- 21838.
33	LAI W S, HUANG J B, AHUJA N, et al. Deep Laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C. , USA: IEEE Press, 2017: 5835-5843.
34	HUI Z, WANG X M, GAO X B. Fast and accurate single image super-resolution via information distillation network[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C. , USA: IEEE Press, 2018: 723-731.
35	LI W B, ZHOU K, QI L, et al. LAPAR: linearly-assembled pixel-adaptive regression network for single image super-resolution and beyond[EB/OL]. [2023-05-22]. http://arxiv.org/abs/2105.10422v1.
36	SUN L, PAN J S, TANG J H. ShuffleMixer: an efficient ConvNet for image super-resolution[EB/OL]. [2023-05-22]. http://arxiv.org/abs/2205.15175v1.
37	KONG F Y, LI M X, LIU S W, et al. Residual local feature network for efficient super-resolution[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Washington D. C. , USA: IEEE Press, 2022: 765-775.

[1]	胡庆. 多尺度融合与双输出U-Net网络的行人重识别[J]. 计算机工程, 2024, 50(6): 102-109.
[2]	孙帮勇, 马铭, 于涛. 基于区域特征强化的多尺度伪装目标检测方法[J]. 计算机工程, 2024, 50(5): 209-219.
[3]	谢新林, 尹东旭, 张涛源, 谢刚. 基于注意力机制的多尺度融合人群计数算法[J]. 计算机工程, 2024, 50(3): 290-297.
[4]	范文卓, 吴涛, 许俊平, 李庆庆, 张建林, 李美惠, 魏宇星. 基于多分辨率特征融合的任意尺度图像超分辨率重建[J]. 计算机工程, 2023, 49(9): 217-225.
[5]	丁子轩, 俞雷, 张娟, 李想, 王新宇. 基于深度残差自适应注意力网络的图像超分辨率重建[J]. 计算机工程, 2023, 49(5): 231-238.
[6]	陈文轩, 曾碧, 郭植星. 融合多特征与语义图卷积网络的摔倒检测方法[J]. 计算机工程, 2023, 49(5): 277-285,294.
[7]	李培育, 张雅丽. 基于改进SRGAN模型的人脸图像超分辨率重建[J]. 计算机工程, 2023, 49(4): 199-205.
[8]	胡清翔, 饶文碧, 熊盛武. 面向无人机遥感场景的轻量级小目标检测算法[J]. 计算机工程, 2023, 49(12): 169-177.
[9]	李刚, 邵瑞, 周鸣乐, 李敏, 万洪林. 基于注意力的轻量级工业产品缺陷检测网络[J]. 计算机工程, 2023, 49(11): 275-283.
[10]	周静, 钟原, 李平, 杨毅, 马立泰, 张涛. 用于颈椎MRI分割的多尺度特征融合注意力网络模型[J]. 计算机工程, 2023, 49(10): 298-304.
[11]	伍子嘉, 陈航, 彭勇, 宋威. 动态环境下融合轻量级YOLOv5s的视觉SLAM[J]. 计算机工程, 2022, 48(8): 187-195,205.
[12]	黎浩民, 李光平. 基于稀疏神经网络的图像超分辨率重建算法[J]. 计算机工程, 2022, 48(7): 247-253.
[13]	柳聪, 屈丹, 司念文, 魏紫薇. 基于深度可分离卷积的轻量级图像超分辨率重建[J]. 计算机工程, 2022, 48(6): 228-234.
[14]	胡宗承, 周亚同, 史宝军, 何昊. 结合注意力机制与特征融合的静态手势识别算法[J]. 计算机工程, 2022, 48(4): 240-246.
[15]	吴旭, 刘翔, 赵静文. 一种轻量级多尺度融合的图像篡改检测算法[J]. 计算机工程, 2022, 48(2): 224-229,236.

选择文件类型/文献管理软件名称

选择包含的内容