基于深度质量感知和分层特征引导的RGB-D显著性检测

doi:10.19678/j.issn.1000-3428.0064616

摘要/Abstract

摘要： 现有基于融合的RGB-D显著性物体检测方法在对跨模态特征进行融合时忽视了RGB和深度图两模态特征的差异性，跨模态特征融合不均衡的问题使得模型不能充分利用跨模态互补特征，而低质量深度图也会对模型性能带来损害。提出一种基于深度质量感知和分层特征引导的RGB-D显著性物体检测算法。算法分为两个阶段：深度质量感知阶段和分层特征引导阶段。在第一阶段，利用深度质量感知从现有的主流RGB-D显著性物体检测训练数据集中挖掘高质量深度图，对训练集进行增强，提升低质量深度图的质量，减少噪声数据对模型性能的损害；在第二阶段，利用特征引导网络对RGB图和深度图进行分层自适应权重动态融合，在有效增加融合效率的同时增强跨模态融合的感知能力。在基准数据集NJUD、NLPR、SSD、STEREO和SIP上的实验结果表明，相比于SSF、CDNet、D3Net、DASNet等方法，该算法能够大幅提升深度图质量，其中在NLPR数据集上F-Measure值为0.934，MAE仅为0.020，综合性能优于其他相关SOTA方法，证明了先挖掘高质量深度图再进行跨模态自适应动态融合算法的有效性。

关键词: 深度质量感知, 特征引导, 跨模态融合, 分层融合, RGB-D显著性检测

Abstract: Existing fusion-based RGB-D saliency object detection methods ignore the differences between RGB and depth map features when fusing cross-modal features.The problems from fusing unbalanced cross-modal features makes the model insufficiently leverage cross-modal complementary features.Moreover，low-quality depth maps can hurt model performance.This paper proposes an RGB-D salient object detection algorithm based on depth quality perception and hierarchical feature guidance.The algorithm is divided into two stages：depth quality perception stage and hierarchical feature guidance stage.In the first stage，depth quality perception is used to mine high-quality depth maps from the existing mainstream RGB-D salient object detection training data sets to enhance the training sets.This process significantly improves the quality of low-quality depth maps and reduces the damage of noise data on model performance.In the second stage，the feature-guidance network is used to perform hierarchical adaptive weight dynamic fusion of the RGB and depth map，which effectively increases the fusion efficiency and enhances the cross-modality fusion perception.The experimental results on five benchmark datasets（NJUD，NLPR，SSD，STEREO，and SIP） show that the proposed algorithm significantly improves the depth map quality compared to methods such as SSF，CDNet，D3Net，and DASNet.Moreover，on the NLPR dataset，the F-Measure value is 0.934，whereas the MAE is only 0.020.The comprehensive performance is better than other related SOTA methods，proving the effectiveness of the proposed algorithm in first mining high-quality depth maps and then performing cross-modal adaptive dynamic fusion.

Key words: depth quality perception, feature guidance, cross-modal fusion, hierarchical fusion, RGB-D saliency detection

中图分类号:

TP391

宋梦柯, 郑元超, 陈程立诏. 基于深度质量感知和分层特征引导的RGB-D显著性检测[J]. 计算机工程, 2023, 49(5): 255-261,268.

SONG Mengke, ZHENG Yuanchao, CHEN Chenglizhao. RGB-D Saliency Detection via Depth Quality Perception and Hierarchical Feature Guidance[J]. Computer Engineering, 2023, 49(5): 255-261,268.

https://www.ecice06.com/CN/Y2023/V49/I5/255

图/表 8

20230515190846

20230515190850

20230515190854

20230515190858

20230515190902

20230515190905

20230515190909

20230515190912

参考文献

[1] LEE H,KIM D.Salient region-based online object tracking[C]//Proceedings of IEEE Winter Conference on Applications of Computer Vision.Washington D.C.,USA:IEEE Press,2018:1170-1177.
[2] XU K,BA J L,KIROS R,et al.Show,attend and tell:neural image caption generation with visual attention[C]//Proceedings of the 32nd International Conference on Machine Learning.New York,USA:ACM Press,2015:2048-2057.
[3] WANG W G,SHEN J B,LING H B.A deep network solution for attention and aesthetics aware photo cropping[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,41(7):1531-1544.
[4] HE J F,FENG J Y,LIU X L,et al.Mobile product search with bag of hash bits and boundary reranking[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2012:3005-3012.
[5] LIU Z,SHI S,DUAN Q,et al.Salient object detection for RGB-D image by single stream recurrent convolution neural network[J].Neurocomputing,2019,363:46-57.
[6] REN J Q,GONG X J,LU Y,et al.Exploiting global priors for RGB-D saliency detection[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2015:25-32.
[7] DING Y,LIU Z,HUANG M,et al.Depth-aware saliency detection using convolutional neural networks[J].Journal of Visual Communication and Image Representation,2019,61:1-9.
[8] CHEN H,LI Y F.Progressively complementarity-aware fusion network for RGB-D salient object detection[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:3051-3060.
[9] ZHAO J X,CAO Y,FAN D P,et al.Contrast prior and fluid pyramid integration for RGBD salient object detection[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2020:3922-3931.
[10] ZHAO J W,ZHAO Y F,LI J,et al.Is depth really necessary for salient object detection?[C]//Proceedings of the 28th ACM International Conference on Multimedia.New York,USA:ACM Press,2020:1745-1754.
[11] ZHANG W B,JI G P,WANG Z,et al.Depth quality-inspired feature manipulation for efficient RGB-D salient object detection[C]//Proceedings of the 29th ACM International Conference on Multimedia.New York,USA:ACM Press,2021:731-740.
[12] JIN W D,XU J,HAN Q,et al.CDNet:complementary depth network for RGB-D salient object detection[J].IEEE Transactions on Image Processing:A Publication of the IEEE Signal Processing Society,2021,30:3376-3390.
[13] ZHU C B,LI G.A multilayer backpropagation saliency detection algorithm and its applications[J].Multimedia Tools and Applications,2018,77(19):25181-25197.
[14] ZHU C B,LI G,WANG W M,et al.An innovative salient object detection using center-dark channel prior[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2018:1509-1515.
[15] WU Z,SU L,HUANG Q M.Cascaded partial decoder for fast and accurate salient object detection[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2020:3902-3911.
[16] HE K M,ZHANG X Y,REN S Q,et al.Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2016:770-778.
[17] FAN D P,LIN Z,ZHANG Z,et al.Rethinking RGB-D salient object detection:models,data sets,and large-scale benchmarks[J].IEEE Transactions on Neural Networks and Learning Systems,2021,32(5):2075-2089.
[18] LI G,ZHU C B.A three-pathway psychobiological framework of salient object detection using stereoscopic technology[C]//Proceedings of IEEE International Conference on Computer Vision Workshops.Washington D.C.,USA:IEEE Press,2018:3008-3014.
[19] JU R,GE L,GENG W J,et al.Depth saliency based on anisotropic center-surround difference[C]//Proceedings of IEEE International Conference on Image Processing.Washington D.C.,USA:IEEE Press,2015:1115-1119.
[20] PENG H,BING L,XIONG W,et al.RGBD salient object detection:a benchmark and algorithms[C]//Proceedings of European Conference on Computer Vision.Berlin,Germany:Springer,2014:92-109.
[21] NIU Y Z,GENG Y J,LI X Q,et al.Leveraging stereopsis for saliency analysis[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2012:454-461.
[22] PIAO Y R,JI W,LI J J,et al.Depth-induced multi-scale recurrent attention network for saliency detection[C]//Proceedings of IEEE/CVF International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2020:7253-7262.
[23] PIAO Y R,RONG Z K,ZHANG M,et al.A2dele:adaptive and attentive depth distiller for efficient RGB-D salient object detection[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2020:9057-9066.
[24] ZHANG M,REN W S,PIAO Y R,et al.Select,supplement and focus for RGB-D saliency detection[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2020:3469-3478.
[25] LIU Z Y,WANG Y,TU Z Z,et al.TriTransNet:RGB-D salient object detection with a triplet transformer embedding network[C]//Proceedings of the 29th ACM International Conference on Multimedia.New York,USA:ACM Press,2021:4481-4490.
[26] JI W,LI J J,ZHANG M,et al.Accurate RGB-D salient object detection via collaborative learning[C]//Proceedings of European Conference on Computer Vision.Berlin,Germany:Springer,2020:52-69.
[27] ZHANG W B,JI G P,WANG Z,et al.Depth quality-inspired feature manipulation for efficient RGB-D salient object detection[C]//Proceedings of the 29th ACM International Conference on Multimedia.New York,USA:ACM Press,2021:731-740.
[28] ZHAO Y F,ZHAO J W,LI J,et al.RGB-D salient object detection with ubiquitous target awareness[J].IEEE Transactions on Image Processing,2021,30:7717-7731.

选择文件类型/文献管理软件名称

选择包含的内容