Super-Resolution Image Reconstruction Algorithm Based on Feedback Mechanism

doi:10.19678/j.issn.1000-3428.0059901

Abstract

Abstract: Although existing super-resolution image reconstruction methods make full use of high-performance deep learning models, they ignore the feedback mechanism that is ubiquitous in the human visual system.This paper proposes a super-resolution image reconstruction algorithm based on a feedback mechanism.The feedback mechanism is implemented by using the hidden states in a recurrent neural network with constraints.For the mechanism, a feedback module is designed to process the feedback connections between networks, and generate more a more persuasive high-level representation that provides more contextual information, which helps high-resolution image reconstruction from low-resolution images.At the same time, a feedback network with a strong ability of early image reconstruction is built.It can gradually generate the final high-resolution image.Furthermore, to address the detail loss of low-resolution images caused by multiple types of degradation, a curriculum learning strategy is introduced to make the network applicable to more complex tasks and improve its robustness.The experimental results show that the proposed algorithm effectively improves the accuracy of super-resolution image reconstruction.Its PSNR value is increased by about 0.5 compared with SRCNN, VDSR, RDN and other algorithms.

Key words: super-resolution image reconstruction, human visual system, deep learning, feedback mechanism, curriculum learning strategy

摘要： 现有的图像超分辨率重建方法充分利用了强大的深度学习模型，但忽略了人类视觉系统中普遍存在的反馈机制。提出一种新型图像超分辨率重建算法，通过具有约束条件的递归神经网络中包含的隐藏状态实现反馈机制，旨在处理网络间的反馈连接并生成更具说服力的高级表示形式，提供更多的上下文信息，从而帮助低分辨率图像完成高分辨率图像的重建。此外，具有较强早期图像重建能力的反馈网络可逐步生成最终的高分辨率图像。为解决低分辨率图像因多种类型的退化而导致的细节损失问题，引入课程学习策略，使网络适用于更复杂的任务，提升模型的鲁棒性。实验结果表明，该算法能有效提升图像超分辨率重建的准确性，与SRCNN、VDSR、RDN等算法相比，其PSNR值最高提升了7.15 dB。

关键词: 图像超分辨率重建, 人类视觉系统, 深度学习, 反馈机制, 课程学习策略

CLC Number:

TP391.41

LOU Xinjie, LI Xiaoxin, LIU Zhiyong. Super-Resolution Image Reconstruction Algorithm Based on Feedback Mechanism[J]. Computer Engineering, 2022, 48(2): 261-267.

楼鑫杰, 李小薪, 刘志勇. 基于反馈机制的图像超分辨率重建算法[J]. 计算机工程, 2022, 48(2): 261-267.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0059901

http://www.ecice06.com/EN/Y2022/V48/I2/261

Figures/Tables 15

References

[1] ZHANG L, WU X.An edge-guided image interpolation algorithm via directional filtering and data fusion[J].IEEE Transactions on Image Processing, 2006, 15(8):2226-2238.
[2] ZHANG K, GAO X, TAO D, et al.Single image super-resolution with non-local means and steering kernel regression[J].IEEE Transactions on Image Processing, 2012, 21(11):4544-4556.
[3] 苏衡, 周杰, 张志浩.超分辨率图像重建方法综述[J].自动化学报, 2013, 39(8):1202-1213. SU H, ZHOU J, ZHANG Z H.A review of super-resolution image reconstruction methods[J].Journal of Automatica Sinica, 2013, 39(8):1202-1213.(in Chinese)
[4] 苏秉华, 金伟其, 牛丽红, 等.超分辨率图像复原及其进展[J].光学技术, 2001, 27(1):6-9. SUN B C, JIN W Q, NIU L H.Super-resolution image restoration and its progress[J].Optical Technique, 2001, 27(1):6-9.(in Chinese)
[5] 浦剑, 张军平.基于词典学习和稀疏表示的超分辨率方法[J].模式识别与人工智能, 2010, 23(3):335-340. PU J, ZHANG J P.Super-resolution method based on dictionary learning and sparse representation[J].Pattem Recognition and Aitificial Intelligence, 2010, 23(3):335-340.(in Chinese)
[6] DONG C, LOY C C, HE K, et al.Learning a deep convolutional network for image super-resolution[C]//Proccedings of European Conference on Computer Vision.Berlin, Germay:Springer, 2014:184-199.
[7] LIAO Q, POGGIO T.Bridging the gaps between residual learning, recurrent neural networks and visual cortex[EB/OL].[2020-10-01].https://www.researchgate.net/publication/301876854_Bridging_the_Gaps_Between_Residual_Learning_Recurrent_Neural_Networks_and_Visual_Cortex.
[8] LUGMAYR A, DANELLJAN M, TIMOFTE R.Ntire 2020 challenge on real-world image super-resolution:methods and results[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:494-495.
[9] KIM J, KWON L J, MU L K.Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:1637-1645.
[10] HAN W, CHANG S, LIU D, et al.Image super-resolution via dual-state recurrent networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:1654-1663.
[11] HUPÉ J M, JAMES A C, PAYNE B R, et al.Cortical feedback improves discrimination between figure and background by V1, V2 and V3 neurons[J].Nature, 1998, 394(6695):784-787.
[12] GILBERT C D, SIGMAN M.Brain states:top-down influences in sensory processing[J].Neuron, 2007, 54(5):677-696.
[13] STOLLENGA M F, MASCI J, GOMEZ F, et al.Deep networks with internal selective attention through feedbackconnections[EB/OL].[2020-10-01].https://www.researchgate.net/publication/263891809_Deep_Networks_with_Internal_Selective_Attention_through_Feedback_Connections.
[14] ZAMIR A R, WU T L, SUN L, et al.Feedback networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:1308-1317.
[15] LEDIG C, THEIS L, HUSZÁR F, et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:4681-4690.
[16] HE K, ZHANG X, REN S, et al.Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:770-778.
[17] HUANG G, LIU Z, VAN D M L, et al.Densely connected convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:4700-4708.
[18] MOSINSKA A, MARQUEZ N P, KOZINSKI M, et al.Beyond the pixel-wise loss for topology-aware delineation[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:3136-3145.
[19] HARIS M, SHAKHNAROVICH G, UKITA N.Deep back-projection networks for super-resolution[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:1664-1673.
[20] PENTINA A, SHARMANSKA V, LAMPERT C H.Curriculum learning of multiple tasks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:5492-5500.
[21] GAO R, GRAUMAN K.On-demand learning for deep image restoration[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2017:1086-1095.
[22] AGUSTSSON E, TIMOFTE R.Ntire 2017 challenge on single image super-resolution:dataset and study[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:126-135.
[23] ZHANG K, ZUO W, GU S, et al.Learning deep CNN denoiser prior for image restoration[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:3929-3938.
[24] ZHANG K, ZUO W, ZHANG L.Learning a single convolutional super-resolution network for multiple degradations[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:3262-3271.
[25] ZHANG Y, TIAN Y, KONG Y, et al.Residual dense network for image super-resolution[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:2472-2481.

Please choose a citation manager

Content to export