Visual SLAM Method Based on Semantic Segmentation in Dynamic Scenes

doi:10.19678/j.issn.1000-3428.0067370

Abstract

Abstract:

A semantic visual SLAM algorithm based on an improved semantic segmentation network DeepLabv3plus and multiview geometry is designed to address the issues of poor robustness and susceptibility to interference from dynamic objects in visual Synchronous Localization And Map (SLAM) construction in dynamic scenes. Based on the semantic segmentation network DeepLabv3plus, a lightweight convolutional network MobileNetV2 is used for feature extraction, and depthwise separable convolutions are used instead of standard convolutions in the Atrous Spatial Pyramid Pooling (ASPP) module. Simultaneously, an attention mechanism is introduced to propose an improved semantic segmentation network DeepLabv3plus. Combining the improved semantic segmentation network DeepLabv3plus with multiview geometry, a dynamic point detection method is proposed to enhance the robustness of visual SLAM in dynamic scenes. On this basis, a three-dimensional semantic static map containing both semantic and geometric information is constructed. The experimental results on the TUM dataset demonstrate that compared with ORB-SLAM2, the highest Root Mean Square Error (RMSE) and Standard Deviation (SD) values increased by more than 98% and 97%, respectively.

Key words: DeepLabv3plus network, visual Synchronous Localization And Map (SLAM), multiview geometry, dynamic scenes, semantic map

摘要：

针对在动态场景下视觉同步定位与建图（SLAM）鲁棒性差、定位与建图精度易受动态物体干扰的问题，设计一种基于改进DeepLabv3plus与多视图几何的语义视觉SLAM算法。以语义分割网络DeepLabv3plus为基础，采用轻量级卷积网络MobileNetV2进行特征提取，并使用深度可分离卷积代替空洞空间金字塔池化模块中的标准卷积，同时引入注意力机制，提出改进的语义分割网络DeepLabv3plus。将改进后的语义分割网络DeepLabv3plus与多视图几何结合，提出动态点检测方法，以提高视觉SLAM在动态场景下的鲁棒性。在此基础上，构建包含语义信息和几何信息的三维语义静态地图。在TUM数据集上的实验结果表明，与ORB-SLAM2相比，该算法在高动态序列下的绝对轨迹误差的均方根误差值和标准差（SD）值最高分别提升98%和97%。

关键词: DeepLabv3plus网络, 视觉同步定位与建图, 多视图几何, 动态场景, 语义地图

Xiaoying DU, Qingni YUAN, Jianyou QI, Chen WANG, Feilong DU, Ao REN. Visual SLAM Method Based on Semantic Segmentation in Dynamic Scenes[J]. Computer Engineering, 2024, 50(3): 242-249.

杜晓英, 袁庆霓, 齐建友, 王晨, 杜飞龙, 任澳. 动态场景下基于语义分割的视觉SLAM方法[J]. 计算机工程, 2024, 50(3): 242-249.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0067370

https://www.ecice06.com/EN/Y2024/V50/I3/242

Figures/Tables 16

Fig.1 Overall framework of visual SLAM

Fig.2 Structure of improved DeepLabv3plus network

Fig.3 Schematic diagram of multiview geometry principles

Fig.4 Comparison of segmentation results of DeepLabv3plus before and after improvement

Fig.5 Comparison of x, y, and z-axis displacements among different algorithms in the fr3/w/rpy sequence

Fig.6 Comparison of attitude angles among different algorithms in fr3/w/rpy sequences

Fig.7 Trajectories among different algorithms under high dynamic sequence fr3/w/xyz

Fig.8 Real motion trajectory of the mobile robot

Fig.9 Feature extraction in the real scene

Fig.10 Three-dimensional trajectory comparison between the proposed algorithm and ORB-SLAM2

Fig.11 fr3/w/static subsequence

Fig.12 Point cloud and octree maps generated by the proposed algorithm

References 26

1	陈首彬. 激光LiDAR/视觉融合的SLAM(LV-SLAM)关键技术研究. 测绘学报, 2023, 52(1): 169. URL
	CHEN S B. Research on key technologies of laser LiDAR/visual fusion SLAM(LV-SLAM). Journal of Surveying and Mapping, 2023, 52(1): 169. URL
2	LIU Z, SHI D, LI R H, et al. PLC-VIO: visual-inertial odometry based on point-line constraints. IEEE Transactions on Automation Science and Engineering, 2022, 19(3): 1880- 1897. doi: 10.1109/TASE.2021.3077026
3	MUR-ARTAL R, TARDOS J D. ORB-SLAM2: an open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Transactions on Robotics, 2017, 33(5): 1255- 1262. doi: 10.1109/TRO.2017.2705103
4	ENGEL J, SCHÖPS T, CREMERS D. LSD-SLAM: large-scale direct monocular SLAM[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 834-849.
5	ENGEL J, KOLTUN V, CREMERS D. Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(3): 611- 625. doi: 10.1109/TPAMI.2017.2658577
6	张慧娟, 方灶军, 杨桂林. 动态环境下基于线特征的RGB-D视觉里程计. 机器人, 2019, 41(1): 75- 82. URL
	ZHANG H J, FANG Z J, YANG G L. RGB-D visual odometer in dynamic environments using line features. Robot, 2019, 41(1): 75- 82. URL
7	杨世强, 范国豪, 白乐乐, 等. 基于几何约束的室内动态环境视觉SLAM. 计算机工程与应用, 2021, 57(16): 203- 212. doi: 10.3778/j.issn.1002-8331.2005-0158
	YANG S Q, FAN G H, BAI L L, et al. Geometric constraints-based visual SLAM under dynamic indoor environment. Computer Engineering and Applications, 2021, 57(16): 203- 212. doi: 10.3778/j.issn.1002-8331.2005-0158
8	DAI W C, ZHANG Y, LI P, et al. RGB-D SLAM in dynamic environments using point correlations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 373- 389. doi: 10.1109/TPAMI.2020.3010942
9	YU C, LIU Z X, LIU X J. DS-SLAM: a semantic visual SLAM towards dynamic environments[C]//Proceedings of International Conference on Intelligent Robots and Systems. Washington D. C., USA: IEEE Press, 2018: 1168-1174.
10	BESCOS B, FÁCIL J M, CIVERA J, et al. DynaSLAM: tracking, mapping and inpainting in dynamic scenes. IEEE Robotics and Automation Letters, 2018, 3(4): 4076- 4083. doi: 10.1109/LRA.2018.2860039
11	ZHONG F W, WANG S, Z ZHANG Z Q. Detect-SLAM: making object detection and SLAM mutually beneficial[C]//Proceedings of IEEE Winter Conference on Applications of Computer Vision. Washington D. C., USA: IEEE Press, 2018: 1001-1010.
12	LIU W, ANGUELOV D, ERHAN D. SSD: single shot multibox detector[EB/OL]. [2023-03-01]. https://link.springer.com/content/pdf/10.1007/978-3-319-46448-0_2.pdf.
13	XIAO L H, WANG J G, QIU X S, et al. Dynamic-SLAM: semantic monocular visual localization and mapping based on deep learning in dynamic environment. Robotics and Autonomous Systems, 2019, 117, 1- 16. doi: 10.1016/j.robot.2019.03.012
14	HE K M, GKIOXARI G, DOLLÁR O, et al. Mask R-CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 42(2): 386- 397.
15	HU Z, ZHAO J, LUO Y, et al. Semantic SLAM based on improved DeepLabv3⁺ in dynamic scenarios. IEEE Access, 2022, 10, 21160- 21168. doi: 10.1109/ACCESS.2022.3154086
16	BOTTLENECKS M I R A. MobileNetV2: inverted residuals and linear bottlenecks[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 18-23.
17	ZHANG X, LI J, HUA Z. MRSE-Net: multiscale residuals and SE-attention network for water body segmentation from satellite images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022, 15, 5049- 5064. doi: 10.1109/JSTARS.2022.3185245
18	SHARMA S, KUMAR S. The Xception model: a potential feature extractor in breast cancer histology images classification. ICT Express, 2022, 8(1): 101- 108. doi: 10.1016/j.icte.2021.11.010
19	PAN Z, HOU J, YU L. Optimization RGB-D 3D reconstruction algorithm based on dynamic SLAM. IEEE Transactions on Instrumentation and Measurement, 2023, 72, 1- 13.
20	ZHOU P, LIU Y, MENG Z. PointSLOT: real-time simultaneous localization and object tracking for dynamic environment. IEEE Robotics and Automation Letters, 2023, 8(5): 2645- 2652. doi: 10.1109/LRA.2023.3256919
21	CAI Y Q, ZHOU W J, ZHANG L T, et al. DHFNet: dual-decoding hierarchical fusion network for RGB-thermal semantic segmentation. The Visual Computer, 2023, 6, 1- 11.
22	CHEN L, WANG Y, MIAO Z, et al. Transformer-based imitative reinforcement learning for multi-robot path planning. IEEE Transactions on Industrial Informatics, 2023, 46, 1- 10.
23	XIE H, ZHANG D, WANG J, et al. Semi-direct multimap SLAM system for real-time sparse 3D map reconstruction. IEEE Transactions on Instrumentation and Measurement, 2023, 72, 1- 13.
24	ROSINOL A, LEONARD J J, CARLONE L. Probabilistic volumetric fusion for dense monocular SLAM[C]//Proceedings of Winter Conference on Applications of Computer Vision. Washington D. C., USA: IEEE Press, 2023: 3097-3105.
25	CHENG S, SUN C, ZHANG S, et al. SG-SLAM: a real-time RGB-D visual SLAM toward dynamic scenes with semantic and geometric information. IEEE Transactions on Instrumentation and Measurement, 2023, 72, 1- 12.
26	WANG J, XU M, ZHAO G, et al. Feature- and distribution-based LiDAR SLAM with generalized feature representation and heuristic nonlinear optimization. IEEE Transactions on Instrumentation and Measurement, 2023, 72, 1- 15.

[1]	ZHOU Qinyuan, DENG Yueping, ZHANG Lei, ZHANG Chen, LU Rirong, HU Xianzhe. Dynamic Visual SLAM System Integrating Optical Flow and Multi-View Geometry [J]. Computer Engineering, 2024, 50(5): 250-259.
[2]	YU Yujin,HAN Jun,ZHAO Qingxi,ZHANG Hongmei. Research on Autonomous Learning Inspection Technology Based on IHDR [J]. Computer Engineering, 2019, 45(4): 311-315,320.
[3]	ZHONG Jiang, SONG Juan. Heterogeneous Data Integration Frame Based on Ontology [J]. Computer Engineering, 2011, 37(14): 44-46.
[4]	FANG Yong, ZHENG Xiao-Lin, CHEN De-Ren. Configurable Information Index Method Based on Semantic Mapping [J]. Computer Engineering, 2010, 36(15): 36-39.
[5]	CUI Xue-Bing, ZHANG Dun-Feng, CUI Beng-Fei. Software Pipelining Eliminating Technology in IA-64 Binary Translation [J]. Computer Engineering, 2010, 36(11): 88-89,92.
[6]	WANG Miao; ; ZHAO Rong-cai ; CAI Guo-ming. Elimination Technology of Software Pipelined Codes in IA-64 Binary Translation [J]. Computer Engineering, 2008, 34(16): 44-46.

Please choose a citation manager

Content to export