基于场景模态深度理解网络的单目图像深度理解

doi:10.19678/j.issn.1000-3428.0059554

摘要/Abstract

摘要： 基于深度卷积神经网络的图像处理方法得到的单目深度图像质量远高于传统图像处理方法，但该方法对无用特征的训练易产生误差积累，且基于回归求解的连续深度距离预测精度较低，导致图像深度信息提取不精确、目标边缘模糊与图像细节缺失。提出一种应用于单目彩色图像的场景模态深度理解网络。建立以堆叠沙漏为主框架的网络模型，通过反复进行自下而上和自上而下的特征提取过程融合低层次纹理与高级语义特征，在每层网络训练中结合离散的深度标签和真实深度图像降低深度理解难度，插入误差修正子模块和极大似然译码优化子模块以准确提取深度特征。实验结果表明，该网络获取的深度信息更准确，其在NYUv2数据集上绝对相关误差较ACAN网络降低0.72%，在KITTI数据集上均方相关误差较GASDA网络降低41.28%，与DORN等深度网络相比，其预测的深度图像包含更多细节信息且目标轮廓更清晰。

关键词: 单目深度理解, 场景模态标签, 有序回归, 误差修正, 极大似然译码

Abstract: The monocular depth image quality obtained by the image processing method based on Depth Convolution Neural Network (DCNN) is much higher than that of traditional image processing methods.However,this method is prone to error accumulation in the training of useless features,and the accuracy of continuous depth distance prediction based on regression solution is low,which leads to inaccurate image depth information extraction,blurred target edge and lack of image details.This paper proposes a Scene Modality Depth Understanding Network(SMDUN) for monocular color images.A network model based on stacked hourglass is established.Through repeated bottom-up and top-down processes,low-level texture and high-level semantic features are fused.In each layer of network training,discrete depth tags and real depth images are combined to reduce the difficulty of depth understanding.Error correction sub module and maximum likelihood decoding optimization sub module are inserted to accurately extract depth features.Experimental results show that the network can obtain more accurate depth information,the Absolute Relative Error(AbsRel) of NYUv2 dataset is 0.72% lower than that of ACAN network,and the Mean Squared Relative Error(MSqRel) of KITTI dataset is 41.28% lower than that of GASDA network.Compared with DORN and other depth networks, the predicted depth image contains more detail information and the target contour is clearer.

Key words: monocular depth understanding, scene modality labeling, ordinal regression, error correction, maximum likelihood decoding

中图分类号:

TP391.41

陈扬, 李大威. 基于场景模态深度理解网络的单目图像深度理解[J]. 计算机工程, 2021, 47(2): 268-278.

CHEN Yang, LI Dawei. Monocular Image Depth Understanding Based on Scene Modality Depth Understanding Network[J]. Computer Engineering, 2021, 47(2): 268-278.

http://www.ecice06.com/CN/Y2021/V47/I2/268

图/表 10

20210225093120

20210225093123

20210225093126

20210225093129

20210225093133

20210225093136

20210225093140

20210225093143

20210225093147

20210225093150

参考文献 37

[1]	FANG Y J,MASAKI I,HORN B.Depth-based target segmentation for intelligent vehicles:fusion of radar and binocular stereo[J].IEEE Transactions on Intelligent Transportation Systems,2002,3(3):196-202.
[2]	KAO J Y,TIAN D,MANSOUR H,VETRO A,et al.Moving object segmentation using depth and optical flow in car driving sequences[C]//Proceedings of 2016 IEEE International Conference on Image Processing.Washington D.C.,USA:IEEE Press,2016:11-15.
[3]	BISWAS J,VELOSO M.Depth camera based indoor mobile robot localization and navigation[C]//Proceedings of 2012 IEEE International Conference on Robotics and Automation.Washington D.C.,USA:IEEE Press,2012:1697-1702.
[4]	CUI Jiyun,ZHANG Hao,HAN Hu,et al.Improving 2D face recognition via discriminative face depth estimation[C]//Proceedings of 2018 International Conference on Biometrics.Washington D.C.,USA:IEEE Press,2018:140-147.
[5]	SAXENA A,CHUNG S H,ANDREW N Y.Learning depth from single monocular images[C]//Proceedings of NIPS'05.Canada,USA:NIPS Foundation,2005:1161-1168.
[6]	ZHANG R,TSAI P S,CRYER J E,et al.Shape-from-shading:a survey[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1999,21(8):690-706.
[7]	NAYAR S K,NAKAGAWA Y.Shape from focus[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,1994,16(8):824-831.
[8]	FAVARO P,SOATTO S.A geometric approach to shape from defocus[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2005,27(3):406-417.
[9]	EIGEN D,PUHRSCH C,FERGUS R.Depth map prediction from a single image using a multi-scale deep network[C]//Proceedings of NIPS'14.Canada,USA:NIPS Foundation,2014:2366-2374.
[10]	XIE J Y,GIRSHICK R,FARHADI A.Deep3D:fully automatic 2D-to-3D video conversion with deep convolutional neural networks[C]//Proceedings of 2016 European Conference on Computer Vision.Berlin,Germany:Springer,2016:842-857.
[11]	LIU Fayao,SHEN Chunhua,LIN Guosheng,et al.Learning depth from single monocular images using deep convolutional neural fields[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,38(10):2024-2039.
[12]	CAO Yuanzhouhan,WU Zifeng,SHEN Chunhua.Estimating depth from monocular images as classification using deep fully convolutional residual networks[J].IEEE Transactions on Circuits and Systems for Video Technology,2017,28(11):3174-3182.
[13]	FU Huan,GONG Mingming,WANG Chaohui,et al.Deep ordinal regression network for monocular depth estimation[C]//Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:2002-2011.
[14]	ZHANG Zhenyu,XU Chunyan,YANG Jian,et al.Progressive hard-mining network for monocular depth estimation[J].IEEE Transactions on Image Processing,2018,27(8):3691-3702.
[15]	NEWELL A,YANG K Y,DENG J.Stacked hourglass networks for human pose estimation[C]//Proceedings of 2016 European Conference on Computer Vision.Berlin,Germany:Springer,2016:483-499.
[16]	COVER T M,THOMAS J A.Elements of information theory[J].Publications of the American Statal Association,2006,103(481):429-429.
[17]	LI J,KLEIN R,YAO A.A two-streamed network for estimating fine-scaled depth maps from single RGB images[C]//Proceedings of 2017 IEEE International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2017:3372-3380.
[18]	XU D,RICCI E,OUYANG W L,WANG X G,et al.Multi-scale continuous CRFs as sequential deep networks for monocular depth estimation[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:5354-5362.
[19]	GARG R,KUMAR B V,CARNEIRO G,et al.Unsupervised CNN for single view depth estimation:geometry to the rescue[C]//Proceedings of 2016 European Conference on Computer Vision.Berlin,Germany:Springer,2016:740-756.
[20]	GODARD C,OISIN M A O,BROSTOW G J.Unsupervised monocular depth estimation with left-right consistency[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:270-279.
[21]	ZHAO Shanshan,FU Huan,GONG Mingming,et al.Geometry-aware symmetric domain adaptation for monocular depth estimation[C]//Proceedings of 2019 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2019:9788-9798.
[22]	CHEN Yuru,ZHAO Haitao,HU Zhengwei.Attention-based context aggregation network for monocular depth estimation[EB/OL].[2020-08-10].https://www.researchgate.net/publication/330725950_Attention-based_Context_Aggrega-tion_Network_for_Monocular_Depth_Estimation.
[23]	ISLAM M A,NAHA S J,ROCHAN M,et al.Label refinement network for coarse-to-fine semantic segmentation[EB/OL].[2020-08-10].https://arxiv.org/abs/1703.00551v1.
[24]	ZHANG F H,PRISACARIU V,YANG R G,et al.Ga-Net:guided aggregation net for end-to-end stereo matching[C]//Proceedings of 2019 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2019:185-194.
[25]	EIGEN D,FERGUS R.Predicting depth,surface normals and semantic labels with a common multi-scale convolu-tional architecture[C]//Proceedings of 2015 IEEE Interna-tional Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2015:2650-2658.
[26]	XIE Saining,TU Zhuowen.Holistically-nested edge detection[J].International Journal of Computer Vision,2017,125(5):3-18.
[27]	LIN G S,ANTON M,SHEN C H,et al.RefineNet:multi-path refinement networks for high-resolution semantic segmenta-tion[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:1925-1934.
[28]	LI Dawei,WANG Sifan,TANG Xuesong,et al.Double-stream atrous network for shadow detection[J].Neurocomputing,2020,417(12):167-175.
[29]	NATHAN S,DEREK H,PUSHMEET K,et al.Indoor segmentation and support inference from RGBD images[C]//Proceedings of 2012 European Conference on Computer Vision.Berlin,Germany:Springer,2012:746-760.
[30]	GEIGER A,LENZ P,STILLER C,et al.Vision meets robotics:the KITTI dataset[J].The International Journal of Robotics Research,2013,32(11):1231-1237.
[31]	OLGA R,DENG J,SU H,et al.ImageNet large scale visual recognition challenge[J].International Journal of Computer Vision,2015,115(3):211-252.
[32]	IRO L,CHRISTIAN R,VASILEIOS B,et al.Deeper depth prediction with fully convolutional residual networks[C]//Proceedings of 2016 International Conference on 3D Vision.Washington D.C.,USA:IEEE Press,2016:239-248.
[33]	WANG Peng,SHEN Xiaohui,LIN Zhe,et al.Towards unified depth and semantic prediction from a single image[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2015:2800-2809.
[34]	LI Bo,DAI Yuchao,HE Mingyi.Monocular depth estimation with hierarchical fusion of dilated CNNs and soft-weighted-sum inference[J].Pattern Recognition,2018,83(11):328-339.
[35]	YEVHEN K,JRG S,BASTIAN L.Semi-supervised deep learning for monocular depth map prediction[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:6647-6655.
[36]	XU Dan,WANG Wei,TANG Hao,et al.Structured attention guided convolutional neural fields for monocular depth estimation[C]//Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:3917-3925.
[37]	GAN Yukang,XU Xiangyu,SUN Wenxiu,et al.Monocular depth estimation with affinity,vertical pooling,and label enhancement[C]//Proceedings of 2018 European Conference on Computer Vision.Berlin,Germany:Springer,2018:224-239.

选择文件类型/文献管理软件名称

选择包含的内容