结合整体注意力与分形稠密特征的图像超分辨率重建

doi:10.19678/j.issn.1000-3428.0062978

摘要/Abstract

摘要： 现有单图像超分辨率模型普遍基于卷积神经网络且使用单一尺度的卷积核提取特征信息，容易造成细节信息遗漏并降低网络表征能力。为有效提取高频信息同时提高图像重建性能，提出一种基于整体注意力机制与分形稠密特征增强的图像超分辨率重建模型。在特征增强过程中，级联9个分形稠密特征增强模块，每个模块通过4条分支路径提取和融合多尺度特征，并引入局部稠密跳跃连接传递信息以获取更丰富的细节信息。引入整体注意力机制，从3个维度出发建立特征图之间的关联关系，通过对不同通道、空间和层次的特征进行加权和选择性聚合为特征图分配不同的权重，从而提高模型判别学习能力。在Set5、Set14、BSDS100和Urban100数据集上的实验结果表明，该模型可有效重建纹理细节更丰富的高分辨率图像，重建图像在主观视觉效果与客观评价指标上均优于同类模型，且在图像放大3倍时，峰值信噪比和结构相似性指标最高比MSRN模型提升了0.57 dB和0.007。

关键词: 超分辨率重建, 卷积神经网络, 多尺度特征提取, 注意力机制, 高频信息

Abstract: Most current single-image super-resolution models are based on Convolutional Neural Networks (CNN) that use a single-scale convolution kernel to extract feature information, resulting in the missing details and reduced network representation ability.This study proposes an image super-resolution reconstruction model based on a holistic attention mechanism and fractal density feature enhancement to effectively extract high-frequency information and improve reconstruction performance. During the feature enhancement process, the proposed model cascades nine Fractal Dense Feature Enhancement(FDFE) modules.Each module extracts and fuses multiscale features through four branch paths and introduces local dense jump connections to transfer information for enhanced, detailed information.Next, the proposed model uses a holistic attention mechanism to establish the correlation between feature maps in three dimensions and assigns different weights to feature maps by weighting and selectively aggregating features from different channels, spaces, and layer to improve the discrimination learning ability of the model.The experimental results on the Set5, Set14, BSDS100, and Urban100 datasets show that the proposed model can effectively reconstruct high-resolution images with richer texture details.It is superior to many similar models in terms of subjective visual effects and objective evaluation indicators.When the image is magnified three times, the Peak Signal-to-Noise Ratio(PSNR) and Structural Similarity(SSIM) indexes of the proposed model are up to 0.57 dB and 0.007 higher than the Multi-Scale Residual Network(MSRN) model.

Key words: super-resolution reconstruction, Convolutional Neural Networks(CNN), multi-scale feature extraction, attention mechanism, high-frequency information

中图分类号:

TP391.4

陈乔松, 蒲柳, 张羽, 孙开伟, 邓欣, 王进. 结合整体注意力与分形稠密特征的图像超分辨率重建[J]. 计算机工程, 2022, 48(11): 207-214,223.

CHEN Qiaosong, PU Liu, ZHANG Yu, SUN Kaiwei, DENG Xin, WANG Jin. Image Super-Resolution Reconstruction Combining Holistic Attention and Fractal Density Feature[J]. Computer Engineering, 2022, 48(11): 207-214,223.

https://www.ecice06.com/CN/Y2022/V48/I11/207

图/表 12

20230221175009

20230221175012

20230221175015

20230221175019

20230221175022

20230221175026

20230221175030

20230221175033

20230221175036

20230221175039

20230221175042

20230221175046

参考文献

[1] 苏衡, 周杰, 张志浩.超分辨率图像重建方法综述[J].自动化学报, 2013, 39(8):1202-1213. SU H, ZHOU J, ZHANG Z H.Survey of super-resolution image reconstruction methods[J].Acta Automatica Sinica, 2013, 39(8):1202-1213.(in Chinese)
[2] GIORGETTI A, VARRELLA M, CHIANI M.Analysis and performance comparison of different cognitive radio algorithms[C]//Proceedings of the 2nd International Workshop on Cognitive Radio and Advanced Spectrum Management.Washington D.C., USA:IEEE Press, 2009:127-131.
[3] IRANI M, PELEG S.Super resolution from image sequences[C]//Proceedings of the 10th International Conference on Pattern Recognition.Washington D.C., USA:IEEE Press, 1990:115-120.
[4] KIM K I, KWON Y.Example-based learning for single-image super-resolution[C]//Proceedings of Joint Pattern Recognition Symposium.Berlin, Germany:Springer, 2008:456-465.
[5] WANG S L, ZHANG L, LIANG Y, et al.Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis[C]//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2012:2216-2223.
[6] JOSEPH P J, VASWANI K, THAZHUTHAVEETIL M J.Construction and use of linear regression models for processor performance analysis[C]//Proceedings of the 12th International Symposium on High-Performance Computer Architecture.Washington D.C., USA:IEEE Press, 2006:99-108.
[7] LI Z, GUO J, XU W C, et al.Parameter extraction for equivalent circuit of common mode choke based on deep neural network, random forest tree and extreme gradient boosting algorithm[C]//Proceedings of Photonics & Electromagnetics Research Symposium.Washington D.C., USA:IEEE Press, 2019:2296-2304.
[8] 连逸亚, 吴小俊.超深卷积神经网络的图像超分辨率重建研究[J].计算机工程, 2019, 45(1):217-220. LIAN Y Y, WU X J.Research on image super-resolution reconstruction of super deep convolutional neural network[J].Computer Engineering, 2019, 45(1):217-220.(in Chinese)
[9] DONG C, LOY C C, HE K M, et al.Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2):295-307.
[10] DONG C, LOY C C, TANG X O.Accelerating the super-resolution convolutional neural network[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2016:391-407.
[11] KIM J, LEE J K, LEE K M.Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:1646-1654.
[12] GHIFARY M, KLEIJN W B, ZHANG M J, et al.Deep reconstruction-classification networks for unsupervised domain adaptation[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2016:597-613.
[13] MAO X, SHEN C, YANG Y B.Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections[J].Advances in Neural Information Processing Systems, 2016(29):2802-2810.
[14] LEDIG C, THEIS L, HUSZÁR F, et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:105-114.
[15] LI J C, FANG F M, MEI K F, et al.Multi-scale residual network for image super-resolution[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2018:527-542.
[16] WINEY B, SHARP G, BUSSIÈRE M.A fast double template convolution isocenter evaluation algorithm with subpixel accuracy[J].Medical Physics, 2011, 38(1):223-227.
[17] HUANG G, LIU Z, VAN DER MAATEN L, et al.Densely connected convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:2261-2269.
[18] NIU B, WEN W L, REN W Q, et al.Single image super-resolution via a holistic attention network[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2020:191-207.
[19] LI Y Y, LI Y L, GU Y F.Channel-wise spatial attention with spatiotemporal heterogeneous framework for action recognition[C]//Proceedings of the 6th International Conference on Computing and Artificial Intelligence.Berlin, Germany:Springer, 2020:334-338.
[20] 张海涛, 张梦.引入通道注意力机制的SSD目标检测算法[J].计算机工程, 2020, 46(8):264-270. ZHANG H T, ZHANG M.SSD target detection algorithm with channel attention mechanism[J].Computer Engineering, 2020, 46(8):264-270.(in Chinese)
[21] MUDIGERE D, NAUMOV M, SPISAK J, et al.Building recommender systems with PyTorch[C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.New York, USA:ACM Press, 2020:3525-3526.
[22] TIMOFTE R, AGUSTSSON E, GOOL L V, et al.NTIRE 2017 challenge on single image super-resolution:methods and results[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops.Washington D.C., USA:IEEE Press, 2017:1110-1121.
[23] SMITH D C.Super-resolution of text images through neighbor embedding[C]//Proceedings of the 13th IASTED International Conference on Signal and Image Processing.Calgary, Canada:ACTA Press, 2011:19-26.
[24] 彭亚丽, 张鲁, 张钰, 等.基于深度反卷积神经网络的图像超分辨率算法[J].软件学报, 2018, 29(4):926-934. PENG Y L, ZHANG L, ZHANG Y, et al.Deep deconvolution neural network for image super-resolution[J].Journal of Software, 2018, 29(4):926-934.(in Chinese)
[25] MARTIN D, FOWLKES C, TAL D, et al.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings of the 8th IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2001:416-423.
[26] HUANG J B, SINGH A, AHUJA N.Single image super-resolution from transformed self-exemplars[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:5197-5206.
[27] FENG J, WANG Z W, ZHA M, et al.Flower recognition based on transfer learning and Adam deep learning optimization algorithm[C]//Proceedings of 2019 International Conference on Robotics, Intelligent Control and Artificial Intelligence.Washington D.C., USA:IEEE Press, 2019:598-604.
[28] FANG Z, XU X X, LI X, et al.SPGD algorithm optimization based on Adam optimizer[C]//Proceedings of AOPC'20.Beijing, China:[s.n.], 2020:677-684.
[29] HUYNH-THU Q, GHANBARI M.Scope of validity of PSNR in image/video quality assessment[J].Electronics Letters, 2008, 44(13):800-810.
[30] LIU D, LI Y C, CHEN S J.No-reference remote sensing image quality assessment based on the region of interest and structural similarity[C]//Proceedings of the 2nd International Conference on Advances in Image Processing.Berlin, Germany:Springer, 2018:64-67.
[31] KEYS R.Cubic convolution interpolation for digital image processing[J].IEEE Transactions on Acoustics, Speech, and Signal Processing, 1981, 29(6):1153-1160.
[32] LAI W S, HUANG J B, AHUJA N, et al.Deep Laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:5835-5843.
[33] AHN N, KANG B, SOHN K A.Fast, accurate, and lightweight super-resolution with cascading residual network[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2018:256-272.
[34] HUI Z, GAO X B, YANG Y C, et al.Lightweight image super-resolution with information multi-distillation network[C]//Proceedings of the 27th ACM International Conference on Multimedia.New York, USA:ACM Press, 2019:2024-2032.
[35] 胡晓辉, 张建国.基于改进卷积神经网络的图像超分辨率算法研究[J].计算机应用研究, 2020, 37(3):947-950, 956. HU X H, ZHANG J G.Research on image super-resolution algorithm based on improved convolutional neural network[J].Application Research of Computers, 2020, 37(3):947-950, 956.(in Chinese)

选择文件类型/文献管理软件名称

选择包含的内容