结合可变形卷积与双边网格的立体匹配网络

doi:10.19678/j.issn.1000-3428.0063606

摘要/Abstract

摘要： 双目立体匹配被广泛应用于无人驾驶、机器人导航、增强现实等三维重建领域。在基于深度学习的立体匹配网络中采用多尺度2D卷积进行代价聚合，存在对目标边缘处的视差预测鲁棒性较差以及特征提取性能较低的问题。提出将可变形卷积与双边网格相结合的立体匹配网络。通过改进的特征金字塔网络进行特征提取，并将注意力特征增强、注意力机制、Meta-ACON激活函数引入到改进的特征金字塔网络中，以充分提取图像特征并减少语义信息丢失，从而提升特征提取性能。利用互相关层进行匹配计算，获得多尺度3D代价卷，采用2D可变形卷积代价聚合结构对多尺度3D代价卷进行聚合，以解决边缘膨胀问题，使用双边网格对聚合后的低分辨率代价卷进行上采样，经过视差回归得到视差图。实验结果表明，该网络在Scene Flow数据集中的端点误差为0.75，相比AANet降低13.8%，在KITTI2012数据集中3px的非遮挡区域误差率为1.81%，能准确预测目标边缘及小区域处的视差。

关键词: 双目视觉, 立体匹配, 双边网格, 可变形卷积, 注意力机制

Abstract: Binocular stereo matching is widely used in the field of 3D reconstruction for applications such as unmanned driving, robot navigation, and augmented reality.In a stereo matching network based on deep learning, multi-scale 2D convolution is used for cost aggregation;however, this type of network exhibits the problem of poor robustness to disparity prediction at target edge and low performance of feature extraction.Thus, a stereo matching network based on deformable convolution and a bilateral grid is proposed.An improved Feature Pyramid Network(FPN) is used for feature extraction, and attention feature enhancement, attention mechanism, and the Meta-ACON activation function are introduced to fully extract image features and reduce semantic information loss, thereby improving the performance of feature extraction.A cross correlation layer is used for matching the calculation to obtain multi-scale 3D cost volumes. The 2D deformable convolution cost aggregation structure is used to aggregate multi-scale 3D cost volumes to solve the problem of edge expansion.The bilateral grid is used to upsample the aggregated low-resolution cost volumes, and a disparity map is obtained through disparity regression.The experimental results show that the End Point Error(EPE) of the network on the Scene Flow dataset is 0.75, which is 13.8% lower than that of the AANet.The error matching rate of the 3px non-occluded area in the KITTI2012 dataset is 1.81%.It can accurately predict the disparity at target edge and the small area.

Key words: binocular vision, stereo matching, bilateral grid, deformable convolution, attention mechanism

中图分类号:

TP391.4

刘振国, 李钊, 宋滕滕, 何益智. 结合可变形卷积与双边网格的立体匹配网络[J]. 计算机工程, 2022, 48(12): 241-247,254.

LIU Zhenguo, LI Zhao, SONG Tengteng, HE Yizhi. Stereo Matching Network Combining Deformable Convolution and Bilateral Grid[J]. Computer Engineering, 2022, 48(12): 241-247,254.

https://www.ecice06.com/CN/Y2022/V48/I12/241

图/表 10

20230112184755

20230112184758

20230112184801

20230112184805

20230112184808

20230112184811

20230112184815

20230112184818

20230112184821

20230112184825

参考文献

[1] 张一飞, 李新福, 田学东.融合边缘特征的SAD立体匹配算法[J].计算机工程, 2020, 46(4):236-240, 246. ZHANG Y F, LI X F, TIAN X D.SAD stereo matching algorithm combining edge features[J].Computer Engineering, 2020, 46(4):236-240, 246.(in Chinese)
[2] 赵晨园, 李文新, 张庆熙.一种改进的实时半全局立体匹配算法及硬件实现[J].计算机工程, 2021, 47(9):162-170. ZHAO C Y, LI W X, ZHANG Q X.An improved real-time semi-global stereo matching algorithm and its hardware implementation[J].Computer Engineering, 2021, 47(9):162-170.(in Chinese)
[3] 黄彬, 胡立坤, 张宇.基于自适应权重的改进Census立体匹配算法[J].计算机工程, 2021, 47(5):189-196. HUANG B, HU L K, ZHANG Y.Improved census stereo matching algorithm based on adaptive weight[J].Computer Engineering, 2021, 47(5):189-196.(in Chinese)
[4] MAYER N, ILG E, HÄUSSER P, et al.A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:4040-4048.
[5] ALEX K, HAYK M, SAUMITRO S, et al.End-to-end learning of geometry and context for deep stereo regression[C]//Proceedings of International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2017:66-75.
[6] CHANG J R, CHEN Y S.Pyramid stereo matching network[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:5410-5418.
[7] HE K M, ZHANG X Y, REN S Q, et al.Spatial pyramid pooling in deep convolutional networks for visual recognition[EB/OL].[2022-01-09].https://arxiv.org/pdf/1406.4729.pdf.
[8] ZHANG F H, PRISACARIU V, YANG R G, et al.GA-Net:guided aggregation net for end-to-end stereo matching[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA.IEEE Press, 2019:185-194.
[9] XU H F, ZHANG J Y.AANet:adaptive aggregation network for efficient stereo matching[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:1956-1965.
[10] XU B, XU Y H, YANG X L, et al.Bilateral grid learning for stereo matching networks[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2021:12492-12501.
[11] CHEN J W, SYLVAIN P, FREDO D.Real-time edge-aware image processing with the bilateral grid[J].ACM Transactions on Graphics, 2017, 26(3):1-10.
[12] ILG E, MAYER N, SAIKIA T, et al.FlowNet 2.0:evolution of optical flow estimation with deep networks[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:1647-1655.
[13] LIN T Y, DOLLÁR P, GIRSHICK R, et al.Feature pyramid networks for object detection[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:936-944.
[14] GUO C X, FAN B, ZHANG Q, et al.AugFPN:improving multi-scale feature learning for object detection[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:12592-12601.
[15] WOO S, PARK J, LEE J Y, et al.CBAM:convolutional block attention module[EB/OL].[2022-01-09].https://arxiv.org/pdf/1807.06521.pdf.
[16] MA N N, ZHANG X Y, LIU M, et al.Activate or not:learning customized activation[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2021:8028-8038.
[17] ZHAO H S, SHI J P, QI X J, et al.Pyramid scene parsing network[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2021:6230-6239.
[18] ZHANG K, FANG Y Q, MIN D B, et al.Cross-scale cost aggregation for stereo matching[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2014:1590-1597.
[19] MENZ M D, FREEMAN R D.Stereoscopic depth processing in the visual cortex:a coarse-to-fine mechanism[J].Nature Neuroscience, 2003, 6(1):59-65.
[20] XU Q S, TAO W B.Multi-scale geometric consistency guided multi-view stereo[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2019:5478-5487.
[21] DAI J F, QI H Z, XIONG Y W, et al.Deformable convolutional networks[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2017:764-773.
[22] GEIGER A, LENZ P, URTASUN R.Are we ready for autonomous driving?The KITTI vision benchmark suite[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2012:3354-3361.
[23] MENZE M, GEIGER A.Object scene flow for autonomous vehicles[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2012:3061-3070.
[25] WANG Q, SHI S H, ZHENG S Z, et al.FAD-Net:a fast and accurate network for disparity estimation[C]//Proceedings of Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:1-7.
[26] 季顺平, 罗冲, 刘瑾.基于深度学习的立体影像密集匹配方法综述[J].武汉大学学报(信息科学版), 2021, 46(2):193-202. JI S P, LUO C, LIU J.A review of dense stereo image matching methods based on deep learning[J].Geomatics and Information Science of Wuhan University, 2021, 46(2):193-202.(in Chinese)
[27] 郑太雄, 黄帅, 李永福, 等.基于视觉的三维重建关键技术研究综述[J].自动化学报, 2020, 46(4):631-652. ZHENG T X, HUANG S, LI Y F, et al.Key techniques for vision based 3D reconstruction:a review[J].Acta Automatica Sinica, 2020, 46(4):631-652.(in Chinese)<

选择文件类型/文献管理软件名称

选择包含的内容