一种改进的超分辨率轻量化特征融合方法

doi:10.19678/j.issn.1000-3428.0068367

摘要/Abstract

摘要：

图像超分辨率技术可以通过提高图像的分辨率从而有效提升图片的质量和观看的视频体验。然而, 小型嵌入式设备因硬件资源受限难以运行常规模型。为减少模型的参数量以及加快模型运算速度, 提出一种改进的超分辨率轻量化特征融合方法ILFM。设计结构间部分参数共享模块, 使得在参数量基本不变的情况下增强模块的表达能力, 进而增强模型输出图像质量, 设计一种更加轻量的可分离编解码基本模块。在模型中双层网络结构和改进的参数共享方法被设计为1个统一结构。除此之外, 采用通道叠加的图像预处理方式来提取更多的图像特征。在DIV2K和Flickr2K数据集上进行训练, 在Set5和BSDS100等多个基准数据集上进行测试, 实验结果表明, 相较于基准模型IMDN, ILFM在超分辨率系数为2和4且输出更高图片质量的情况下参数量分别降低了63%和61%。对比当前最优的轻量化超分辨率模型, ILFM能够在多个数据集的峰值信噪比(PSNR)和结构相似性(SSIM)上取得平均0.043 78 dB和0.001 3的增长, 具有更优的综合性能。

关键词: 图像超分辨率, 轻量化, 卷积, 参数共享, 编解码

Abstract:

Image super-resolution technology effectively enhances picture quality and improves the viewing experience of videos by increasing the resolution of images. However, it is difficult for small embedded devices to run conventional models due to limited hardware resources. To reduce the number of model parameters and accelerate computation, this study proposes ILFM as an improved super-resolution lightweight feature fusion method. A parameter sharing module between structures is designed to enhance expressive power while maintaining a relatively constant number of parameters, thereby improving the quality of model output. Furthermore, a lighter and separable encoding and decoding basic module is introduced. The dual-layer network structure and improved parameter sharing method are designed as a unified structure in the model, and the image preprocessing method of channel stacking is used to extract additional image features. The network is trained on the DIV2K and Flickr2K datasets and subsequently tested on multiple benchmark datasets such as Set5 and BSDS100. The experimental results show that compared to the IMDN benchmark model, ILFM reduces the number of model parameters by 63% and 61% for super-resolution coefficients of 2 and 4, respectively, as well as improving output image quality. Compared to the current optimal lightweight super-resolution model, ILFM achieves an average increase of 0.043 78 dB and 0.001 3 in Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM), respectively, across multiple datasets, demonstrating superior overall performance.

Key words: image super-resolution, lightweight, convolution, parameter sharing, encode-decode

李志鹏, 陈丹阳, 钟诚. 一种改进的超分辨率轻量化特征融合方法[J]. 计算机工程, 2024, 50(11): 258-265.

LI Zhipeng, CHEN Danyang, ZHONG Cheng. An Improved Super-Resolution Lightweight Feature Fusion Method[J]. Computer Engineering, 2024, 50(11): 258-265.

https://www.ecice06.com/CN/Y2024/V50/I11/258

图/表 9

图1 改进的双层信息蒸馏结构

Fig.1 Structure of improved dual-layer information distillation

图2 常规模型结构

Fig.2 Structure of conventional model

图3 参数共享示意图

Fig.3 Schematic diagram of parameter sharing

图4 波动模块

Fig.4 Fluctuation module

图5 不同模型的参数量及输出图像的PSNR

Fig.5 Parameter quantities and PSNR of output images of different models

图6 当超分辨率系数为4时不同模型的输出结果对比

Fig.6 Comparison of output results of different models when the super-resolution coefficient is 4

参考文献 37

1	杨才东, 李承阳, 李忠博, 等. 深度学习的图像超分辨率重建技术综述. 计算机科学与探索, 2022, 16 (9): 1990- 2010.
	YANG C D, LI C Y, LI Z B, et al. Review of image super-resolution reconstruction algorithms based on deep learning. Journal of Frontiers of Computer Science and Technology, 2022, 16 (9): 1990- 2010.
2	黄健, 赵元元, 郭苹, 等. 深度学习的单幅图像超分辨率重建方法综述. 计算机工程与应用, 2021, 57 (18): 13- 23. doi: 10.3778/j.issn.1002-8331.2102-0257
	HUANG J, ZHAO Y Y, GUO P, et al. Survey of single image super-resolution based on deep learning. Computer Engineering and Applications, 2021, 57 (18): 13- 23. doi: 10.3778/j.issn.1002-8331.2102-0257
3	DONG C, LOY C C, HE K M, et al. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38 (2): 295- 307. doi: 10.1109/TPAMI.2015.2439281
4	张晨晨, 王帅, 王文一, 等. 针对人脸识别卷积神经网络的局部背景区域对抗攻击. 光电工程, 2023, 50 (1): 220266. doi: 10.12086/oee.2023.220266
	ZHANG C C, WANG S, WANG W Y, et al. Adversarial background attacks in a limited area for CNN based face recognition. Opto-Electronic Engineering, 2023, 50 (1): 220266. doi: 10.12086/oee.2023.220266
5	DONG C, LOY C C, TANG X O. Accelerating the super-resolution convolutional neural network[EB/OL]. [2023-08-05]. https://link.springer.com/content/pdf/10.1007/978-3-319-46475-6_25.pdf?pdf=core.
6	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2016: 770-778.
7	ZHANG Y L, TIAN Y P, KONG Y, et al. Residual dense network for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (7): 2480- 2495. doi: 10.1109/TPAMI.2020.2968521
8	KIM J W, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2016: 1-10.
9	LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Washington D. C., USA: IEEE Press, 2017: 1132-1140.
10	LEDIG C, THEIS L, HUSZAR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2017: 1-10.
11	王云涛, 赵蔺, 刘李漫, 等. 基于组-信息蒸馏残差网络的轻量级图像超分辨率重建. 自动化学报, 2024, 50 (10): 1- 16.
	WANG Y T, ZHAO L, LIU L M, et al. G-IDRN: an group-information distillation residual network for lightweight image super-resolution. Acta Automatica Sinica, 2024, 50 (10): 1- 16.
12	MADAAN D, SHIN J, HWANG S J. Adversarial neural pruning with latent vulnerability suppression[C]//Proceedings of International Conference on Machine Learning. New York, USA: [s. n], 2012: 1-10.
13	DONG X Y, YANG Y. Network pruning via Transformable architecture search[EB/OL]. [2023-08-05]. https://arxiv.org/abs/1905.09717v2.
14	WANG Z, WOHLWEND J, LEI T. Structured pruning of large language models[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). New York, USA: [s. n.], 2020: 6151-6162.
15	ZHANG Y, XIANG T, HOSPEDALES T M, et al. Deep mutual learning[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 4320-4328.
16	HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network[EB/OL]. [2023-08-05]. https://www.arxiv.org/abs/1503.02531.
17	XU Z, HSU Y C, HUANG J W. Training shallow and thin networks for acceleration via knowledge distillation with conditional adversarial networks[EB/OL]. [2023-08-05]. https://arxiv.org/pdf/1709.00513.
18	LI Y C, CAO J J, LI Z T, et al. Lightweight single image super-resolution with dense connection distillation network. ACM Transactions on Multimedia Computing, Communications, and Applications, 2021, 17 (1): 1- 17. doi: 10.1145/3414838
19	HUI Z, WANG X M, GAO X B. Fast and accurate single image super-resolution via information distillation network[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 723-731.
20	LIU J, TANG J, WU G S. Residual feature distillation network for lightweight image super-resolution[C]//Proceedings of Europe Conference on Computer Vision. Berlin, Germany: Springer, 2020: 41-55.
21	LI Z Y, LIU Y Q, CHEN X Y, et al. Blueprint separable residual network for efficient image super-resolution[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Washington D. C., USA: IEEE Press, 2022: 832-842.
22	KIM J W, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2016: 1637-1645.
23	TAI Y, YANG J, LIU X M. Image super-resolution via deep recursive residual network[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2017: 2790-2798.
24	BREIMAN L. Bagging predictors. Machine Learning, 1996, 24 (2): 123- 140. URL
25	LIU P, HAN S Z, MENG Z B, et al. Facial expression recognition via a boosted deep belief network[C]//Proceedings of Conference on Computer Vision & Pattern Recognition. Washington D. C., USA: IEEE Press, 2014: 1-10.
26	SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2015: 1-9.
27	SOCHER R, HUVAL B, BHAT B, et al. Convolutional-recursive deep learning for 3D object classification[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2012: 656-664.
28	LIANG M, HU X L. Recurrent convolutional neural network for object recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2015: 3367-3375.
29	TIMOFTE R, AGUSTSSON E, COOL L V, et al. NTIRE 2017 challenge on single image super-resolution: dataset and study[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Washington D. C., USA: IEEE Press, 2017: 1122-1131.
30	BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[EB/OL]. [2023-08-05]. https://www.researchgate.net/profile/Christine-Guillemot-3/publication/260351242_Low-Complexity_Single_Image_Super-Resolution_Based_on_Nonnegative_Neighbor_Embedding/links/5424017c0cf26120b7a70f86/Low-Complexity-Single-Image-Super-Resolution-Based-on-Nonnegative-Neighbor-Embedding.pdf.
31	ZEYDE R, ELAD M, PROTTER M. On single image scale-up using sparse-representations[C]//Proceedings of the 7th International Conference on Curves and Surfaces. Berlin, Germany: Springer, 2010: 711-730.
32	MARTIN D, FOWLKES C, TAL D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings the 8th IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2001: 416-423.
33	HUANG J B, SINGH A, AHUJA N. Single image super-resolution from transformed self-exemplars[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2015: 5197-5206.
34	MATSUI Y, ITO K, ARAMAKI Y, et al. Sketch-based manga retrieval using Manga109 dataset[EB/OL]. [2023-08-05]. https://link.springer.com/content/pdf/10.1007/s11042-016-4020-z.pdf?pdf=core.
35	CHU X, ZHANG B, MA H, et al. Fast, accurate and lightweight super-resolution with neural architecture search[C]//Proceedings of the 25th International Conference on Pattern Recognition(ICPR). Washington D. C., USA: IEEE Press, 2021: 59-64.
36	ZHAO H, KONG X, HE J, et al. Efficient image super-resolution using pixel attention[C]//Proceedings of Europe Conference on Computer Vision. Berlin, Germany: Springer, 2020: 56-72.
37	KONG F Y, LI M X, LIU S W, et al. Residual local feature network for efficient super-resolution[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Washington D. C., USA: IEEE Press, 2022: 765-775.

[1]	王志浩, 钱沄涛. 基于Swin Transformer的双流遥感图像时空融合超分辨率重建[J]. 计算机工程, 2024, 50(9): 33-45.
[2]	李俊俊, 董建刚, 李坤. 基于Kubernetes的集群节能策略研究[J]. 计算机工程, 2024, 50(9): 82-91.
[3]	杨明强, 卢健. 基于跨模态注意力的目标语音提取[J]. 计算机工程, 2024, 50(9): 121-129.
[4]	汤志康, 武毓琦, 李春英, 汤庸. 基于知识图谱卷积网络的学习资源推荐[J]. 计算机工程, 2024, 50(9): 153-160.
[5]	蔡俊民, 梁正友, 孙宇, 陈子奥. 基于可变形三维图卷积的轻量级点云分类研究[J]. 计算机工程, 2024, 50(9): 255-265.
[6]	张鲁, 田春伟, 宋焕生, 刘侍刚. 用于低剂量CT图像去噪的多级双树复小波网络[J]. 计算机工程, 2024, 50(9): 266-275.
[7]	阳金霖, 李朝锋. 融合快速傅里叶卷积的域变换图像去雨滴方法[J]. 计算机工程, 2024, 50(9): 296-303.
[8]	高煜宝, 文志诚. 基于注意力机制的双路解码器图像去噪方法[J]. 计算机工程, 2024, 50(9): 324-332.
[9]	饶日昕, 王怡文, 曾砺志, 童心恬, 赵海涛. 面向废旧电缆检测的轻量化网络模型[J]. 计算机工程, 2024, 50(8): 22-30.
[10]	王蕾, 党时鹏, 潘丰. 基于卷积神经网络的隐匿性旁路预测模型[J]. 计算机工程, 2024, 50(8): 40-49.
[11]	刘锁兰, 王炎, 王洪元, 朱生升. 基于多流语义图卷积网络的人体行为识别[J]. 计算机工程, 2024, 50(8): 64-74.
[12]	闵莉, 董冰洁, 安冬. 基于多注意力机制与跨特征融合的语义分割算法[J]. 计算机工程, 2024, 50(8): 282-289.
[13]	郭伟, 王欣哲, 王江达, 王春艳. 基于卷积调制与空间协作的水下图像增强[J]. 计算机工程, 2024, 50(8): 310-318.
[14]	耿丽丽, 牛保宁. 基于通道相似度熵的卷积神经网络裁剪[J]. 计算机工程, 2024, 50(7): 133-143.
[15]	张洋, 刘畅, 李少青. 基于可控制性度量的图神经网络门级硬件木马检测方法[J]. 计算机工程, 2024, 50(7): 164-173.

选择文件类型/文献管理软件名称

选择包含的内容