基于多模态多级特征聚合网络的光场显著性目标检测

doi:10.19678/j.issn.1000-3428.0061811

摘要/Abstract

摘要： 现有基于深度学习的显著性检测算法主要针对二维RGB图像设计，未能利用场景图像的三维视觉信息，而当前光场显著性检测方法则多数基于手工设计，特征表示能力不足，导致上述方法在各种挑战性自然场景图像上的检测效果不理想。提出一种基于卷积神经网络的多模态多级特征精炼与融合网络算法，利用光场图像丰富的视觉信息，实现面向四维光场图像的精准显著性检测。为充分挖掘三维视觉信息，设计2个并行的子网络分别处理全聚焦图像和深度图像。在此基础上，构建跨模态特征聚合模块实现对全聚焦图像、焦堆栈序列和深度图3个模态的跨模态多级视觉特征聚合，以更有效地突出场景中的显著性目标对象。在DUTLF-FS和HFUT-Lytro光场基准数据集上进行实验对比，结果表明，该算法在5个权威评估度量指标上均优于MOLF、AFNet、DMRA等主流显著性目标检测算法。

关键词: 深度图, 特征融合, 光场, 聚合网络, 显著性目标检测

Abstract: Most existing deep learning based saliency detection algorithms focus on 2D RGB images. However, they fail to take advantage of 3D visual information of scenes.Most light field saliency detection methods are based on hand-crafted features, whose feature representation capacity is insufficient.These issues lead to poor performance in many challenging scene images.To remedy these problems, this paper proposes a multi-modal multi-level feature aggregation network based on convolutional neural network for light field salient object detection.To fully exploit 3D visual information, two stream sub-network are designed in parallel to handle all-focus images and depth maps separately.Moreover, several feature aggregation modules are developed to aggregate multi-level features to detect the salient objects in scene.Moreover, several cross-modal feature fusion modules are designed to fuse multi-modal features from all-focus images, focal stack, and depth maps, which can highlight a salient object by utilizing 3D visual information.Comprehensive experimental comparisons were performed on the DUTLF-FS and HFUT-Lytro light field benchmark datasets, and the results reveal that the algorithm outperforms the mainstream salient target detection algorithms, such as MOLF, AFNet, and DMRA on five authoritative evaluation metrics.

Key words: depth map, feature fusion, light field, aggregation network, Salient Object Detection(SOD)

中图分类号:

TP391.41

王安志, 任春洪, 何淋艳, 杨元英, 欧卫华. 基于多模态多级特征聚合网络的光场显著性目标检测[J]. 计算机工程, 2022, 48(7): 227-233,240.

WANG Anzhi, REN Chunhong, HE Linyan, YANG Yuanying, QU Weihua. Light Field Salient Object Detection Based on Multi-modal Multi-level Feature Aggregation Network[J]. Computer Engineering, 2022, 48(7): 227-233,240.

https://www.ecice06.com/CN/Y2022/V48/I7/227

图/表 7

20221029175202

20221029175206

20221029175210

20221029175214

20221029175217

20221029175220

20221029175224

参考文献

[1] FAN D P, JI G P, SUN G L, et al.Camouflaged object detection[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:2774-2784.
[2] ZHOU W H, LIANG L K, ZHANG H, et al.Scale and orientation aware EPI-patch learning for light field depth estimation[C]//Proceedings of the 24th International Conference on Pattern Recognition.Beijing, China:[s.n.], 2018:2362-2367.
[3] SONG G, LEE K M.Depth estimation network for dual defocused images with different depth-of-field[C]//Proceedings of the 25th IEEE International Conference on Image Processing.Washington D.C., USA:IEEE Press, 2018:1563-1567.
[4] YEUNG H W F, HOU J H, CHEN X M, et al.Light field spatial super-resolution using deep efficient spatial-angular separable convolution[J].IEEE Transactions on Image Processing, 2019, 28(5):2319-2330.
[5] HOU Q B, CHENG M M, HU X W, et al.Deeply supervised salient object detection with short connections[C]//Proceedings of IEEE Conference on Pattern Analysis and Machine Intelligence.Washington D.C., USA:IEEE Press, 2017:815-828.
[6] ZHANG X N, WANG T T, QI J Q, et al.Progressive attention guided recurrent network for salient object detection[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:714-722.
[7] DENG Z J, HU X W, ZHU L, et al.R⊃3;Net:recurrent residual refinement network for saliency detection[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence.Stockholm, Sweden:[s.n.], 2018:684-690.
[8] 花卉.多视觉特征结合有约束简化群优化的显著性目标检测[J].计算机工程, 2015, 41(11):257-262. HUA H.Salient object detection of multi-visual feature combining with constrained simplified swarm optimization[J].Computer Engineering, 2015, 41(11):257-262.(in Chinese)
[9] 李东民, 李静, 梁大川, 等.基于多尺度先验深度特征的多目标显著性检测方法[J].自动化学报, 2019, 45(11):2058-2070. LI D M, LI J, LIANG D C, et al.Multiple salient objects detection using multi-scale prior and deep features[J].Acta Automatica Sinica, 2019, 45(11):2058-2070.(in Chinese)
[10] 张晴, 李云, 李文举, 等.融合深度特征和多核增强学习的显著目标检测[J].中国图象图形学报, 2019, 24(7):1096-1105. ZHANG Q, LI Y, LI W J, et al.Salient object detection via deep features and multiple kernel boosting learning[J].Journal of Image and Graphics, 2019, 24(7):1096-1105.(in Chinese)
[11] WANG A Z, WANG M H.RGB-D salient object detection via minimum barrier distance transform and saliency fusion[J].IEEE Signal Processing Letters, 2017, 24(5):663-667.
[12] HAN J W, CHEN H, LIU N, et al.CNNs-based RGB-D saliency detection via cross-view transfer and multiview fusion[J].IEEE Transactions on Cybernetics, 2018, 48(11):3171-3183.
[13] WANG T T, PIAO Y R, LU H C, et al.Deep learning for light field saliency detection[C]//Proceedings of IEEE/CVF International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2019:8837-8847.
[14] WANG A Z, WANG M H, LI X Y, et al.A two-stage Bayesian integration framework for salient object detection on light field[J].Neural Processing Letters, 2017, 46(3):1083-1094.
[15] WANG A Z, WANG M H, PAN G, et al.Salient object detection with high-level prior based on Bayesian fusion[J].IET Computer Vision, 2017, 11(3):199-206.
[16] ZHANG LI M I, WEI J, et al.Memory-oriented decoder for light field salient object detection[C]//Proceedings of International Conference on Neural Information Processing Systems.Washington D.C., USA:IEEE Press, 2019:896-906.
[17] FAN D P, CHENG M M, LIU J J, et al.Salient objects in clutter:bringing salient object detection to the foreground[C]//Proceedings of European Conference on Computer Vision.Munich, Germany:Springer, 2018:186-202.
[18] LI G B, YU Y Z.Visual saliency detection based on multiscale deep CNN features[J].IEEE Transactions on Image Processing, 2016, 25(11):5012-5024.
[19] ZHAO R, OUYANG W L, LI H S, et al.Saliency detection by multi-context deep learning[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:1265-1274.
[20] LIU N, HAN J W, YANG M H.PiCANet:learning pixel-wise contextual attention for saliency detection[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:3089-3098.
[21] 张青博, 王斌, 崔宁宁, 等.基于注意力机制的规范化矩阵分解推荐算法[J].软件学报, 2020, 31(3):778-793. ZHANG Q B, WANG B, CUI N N, et al.Attention-based regularized matrix factorization for recommendation[J].Journal of Software, 2020, 31(3):778-793.(in Chinese)
[22] 周雨佳, 窦志成, 葛松玮, 等.基于递归神经网络与注意力机制的动态个性化搜索算法[J].计算机学报, 2020, 43(5):812-826. ZHOU Y J, DOU Z C, GE S W, et al.Dynamic personalized search based on RNN with attention mechanism[J].Chinese Journal of Computers, 2020, 43(5):812-826.(in Chinese)
[23] 冯兴杰, 张乐, 曾云泽.基于多注意力CNN的问题相似度计算模型[J].计算机工程, 2019, 45(9):284-290. FENG X J, ZHANG L, ZENG Y Z.Question similarity calculation model based on multi-attention CNN[J].Computer Engineering, 2019, 45(9):284-290.(in Chinese)
[24] WANG L Z, WANG L J, LU H C, et al.Salient object detection with recurrent fully convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(7):1734-1746.
[25] LI G B, YU Y Z.Deep contrast learning for salient object detection[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:478-487.
[26] ZHANG P P, WANG D, LU H C, et al.Amulet:aggregating multi-level convolutional features for salient object detection[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2017:202-211.
[27] WANG W G, LAI Q X, FU H Z, et al.Salient object detection in the deep learning era:an in-depth survey[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(6):3239-3259.
[28] QU L Q, HE S F, ZHANG J W, et al.RGBD salient object detection via deep fusion[J].IEEE Transactions on Image Processing, 2017, 26(5):2274-2285.
[29] CHEN H, LI Y F.Progressively complementarity-aware fusion network for RGB-D salient object detection[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:3051-3060.
[30] CHEN H, LI Y F, SU D.Multi-modal fusion network with multi-scale multi-path and cross-modal interactions for RGB-D salient object detection[J].Pattern Recognition, 2019, 86:376-385.
[31] CHEN H, LI Y F.Three-stream attention-aware network for RGB-D salient object detection[J].IEEE Transactions on Image Processing, 2019, 28(6):2825-2835.
[32] ZHU C B, CAI X, HUANG K, et al.PDNet:prior-model guided depth-enhanced network for salient object detection[C]//Proceedings of International Conference on Multimedia and Expo.Shanghai, China:[s.n.], 2019:199-204.
[33] WANG N N, GONG X J.Adaptive fusion for RGB-D salient object detection[J].IEEE Access, 2019, 7:55277-55284.
[34] PIAO Y R, JI W, LI J J, et al.Depth-induced multi-scale recurrent attention network for saliency detection[C]//Proceedings of IEEE/CVF International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2019:7253-7262.
[35] LI N Y, YE J W, JI Y, et al.Saliency detection on light field[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2014:2806-2813.
[36] LI N Y, SUN B L, YU J Y.A weighted sparse coding framework for saliency detection[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:5216-5223.
[37] ZHANG J, WANG M, GAO J, et al.Saliency detection with a deeper investigation of light field[C]//Proceedings of the 24th International Joint Conference on Artificial Intelligence.Buenos Aires, Argentina:[s.n.], 2015:2212-2218.
[38] ZHANG J, WANG M, LIN L, et al.Saliency detection on light field:a multi-cue approach[J].ACM Transactions on Multimedia Computing, Communications, and Applications, 2017, 13(3):32.
[39] 李爽, 邓慧萍, 朱磊, 等.联合聚焦度和传播机制的光场图像显著性检测[J].中国图象图形学报, 2020, 25(12):2578-2586. LI S, DENG H P, ZHU L, et al.Saliency detection on a light field via the focusness and propagation mechanism[J].Journal of Image and Graphics, 2020, 25(12):2578-2586.(in Chinese)
[40] CHEN Z Y, XU Q Q, CONG R M, et al.Global context-aware progressive aggregation network for salient object detection[J].Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7):10599-10606.
[41] WEI J, WANG S H, HUANG Q M.F⊃3;Net:fusion, feedback and focus for salient object detection[J].Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7):12321-12328.
[42] ZHAO J X, LIU J J, FAN D P, et al.EGNet:edge guidance network for salient object detection[C]//Proceedings of IEEE/CVF International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2019:8778-8787.
[43] ZUANG J T, TANG T, DING Y F, et al.Optimizer:adapting stepsizes by the belief in observed gradients[C]//Proceedings of Conference on Neural Information Processing Systems.Washington D.C., USA:IEEE Press, 2020:136-145.
[44] MARGOLIN R, ZELNIK-MANOR L, TAL A.How to evaluate foreground maps[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2014:248-255.
[45] PERAZZI F, KRÄHENBÜHL P, PRITCH Y, et al.Saliency filters:contrast based filtering for salient region detection[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2012:733-740.
[46] FAN D P, CHENG M M, LIU Y, et al.Structure-measure:a new way to evaluate foreground maps[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2017:4558-4567.
[47] FAN D P, GONG C, CAO Y, et al.Enhanced-alignment measure for binary foreground map evaluation[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence.Stockholm, Sweden:[s.n.], 2018:698-704.
[48] WU Z, SU L, HUANG Q M.Cascaded partial decoder for fast and accurate salient object detection[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2019:3907-3916.

选择文件类型/文献管理软件名称

选择包含的内容