基于鸟瞰图融合的多级旋转等变目标检测网络

doi:10.19678/j.issn.1000-3428.0068696

摘要/Abstract

摘要：

随着自动驾驶系统的发展, 针对道路场景下的3D目标检测受到研究人员的广泛关注。然而, 大多数基于单一传感器或者多传感器融合的目标检测方法未考虑实际道路场景中车辆旋转, 使得捕获的场景同步旋转, 从而导致目标检测性能降低。针对这类问题, 提出一种基于多传感器融合的多级全局旋转等变目标检测网络架构, 以缓解场景旋转造成的目标检测困难, 从而提高目标检测性能。首先, 对体素内部进行各点之间距离编码, 增强局部点云几何信息, 并提取体素的全局旋转等变特征; 其次, 引入图像的语义信息并提取全局旋转等变特征, 进一步提高网络性能; 最后, 将具有旋转等变性的点云和图像信息在鸟瞰图上进行融合, 并嵌入群等变网络提取融合鸟瞰图级全局旋转等变特征。实验结果表明, 该网络架构在nuScenes验证集上达到了68.7%的平均精度均值(mAP)和71.7的nuScenes检测分数(NDS), 以及平均角度误差均值(mAOE)降低到0.288, 相比主流的目标检测方法, 其实现了网络架构本身的旋转等变性并在性能上得到了提升, 此外, 各个组件对于整体网络架构的目标检测性能提升都起到了重要作用。

关键词: 多传感器融合, 体素, 鸟瞰图, 旋转等变, 3D目标检测

Abstract:

With the development of autonomous driving systems, the detection of three-dimensional objects in road scenes has garnered widespread attention. However, most single-sensor or multi-sensor fusion-based object detection methods do not consider the synchronized rotation of the captured scene owing to vehicle movement in real road scenes, which impairs object detection performance. To address such problems, this study proposes a multi-level global rotational equivariant object detection network framework based on multi-sensor fusion to alleviate the difficulty of object detection caused by scene rotation and thereby improve object detection performance. First, the interior of the voxels is encoded by the distance between each point to enhance the local point cloud geometric information and extract the global rotational equivariant features of the voxels. Second, the semantic information of the image is introduced, and global rotational equivariant features are extracted to further improve the network performance. Finally, the point cloud and image information, all with rotational equivariants, are fused on a Bird's-Eye View (BEV) and embedded in a group equivariant network to extract the global rotational equivariant features on the fused BEV level. Experimental results on the nuScenes validation set show that the network architecture achieves a mean Average Precision (mAP) of 68.7% and a nuScenes Detection Score (NDS) of 71.7. Moreover, the mean Average Orientation Error (mAOE) decreases to 0.288. Compared with mainstream object detection methods, the proposed method realizes the rotational equivariance of the network architecture and improves performance. In addition, each component plays an important role in improving the object detection performance of the overall network architecture.

Key words: multi-sensor fusion, voxels, Bird's-Eye View(BEV), rotational equivariance, 3D object detection

刘宏纬, 邵东恒, 杨剑, 魏宪, 李科, 游雄. 基于鸟瞰图融合的多级旋转等变目标检测网络[J]. 计算机工程, 2024, 50(11): 246-257.

LIU Hongwei, SHAO Dongheng, YANG Jian, WEI Xian, LI Ke, YOU Xiong. Multi-Level Rotational Equivariant Object Detection Network Based on BEV Fusion[J]. Computer Engineering, 2024, 50(11): 246-257.

https://www.ecice06.com/CN/Y2024/V50/I11/246

图/表 17

图1 基于鸟瞰图融合的多级旋转等变目标检测网络架构

Fig.1 Multi-level rotational equivariant object detection network architecture based on BEV fusion

图2 体素内部点间距离编码

Fig.2 Distance coding of voxel internal point-to-point

图3 体素全局旋转等变提升过程

Fig.3 Lifting process of voxel global rotational equivariance

图4 体素全局旋转等变多尺度群卷积

Fig.4 Multi-scale group convolution of voxel global rotational equivariance

图5 体素全局旋转等变多尺度群反卷积

Fig.5 Multi-scale group deconvolution of voxel global rotational equivariance

图6 图像全局旋转等变

Fig.6 Image global rotational equivariance

图7 融合鸟瞰图级全局旋转等变

Fig.7 Fused BEV-level global rotational equivariance

图8 基准网络与MEqBEVNet的可视化结果

Fig.8 Visualization results of benchmark network and MEqBEVNet

参考文献 36

1	韩磊, 高永彬, 史志才. 基于稀疏Transformer的雷达点云三维目标检测. 计算机工程, 2022, 48 (11): 104-110, 144. URL
	HAN L, GAO Y B, SHI Z C. Three-dimensional object detection of radar point cloud based on sparse Transformer. Computer Engineering, 2022, 48 (11): 104-110, 144. URL
2	窦允冲, 侯进, 曾雷鸣, 等. 基于反馈机制与空洞卷积的道路小目标检测网络. 计算机工程, 2023, 49 (1): 287- 294. URL
	DOU Y C, HOU J, ZENG L M, et al. Road small target detection network based on feedback mechanism and dilated convolution. Computer Engineering, 2023, 49 (1): 287- 294. URL
3	LI Z Q, WANG W H, LI H Y, et al. BEVFormer: learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2022: 1-18.
4	雷嘉铭, 俞辉, 夏羽, 等. 基于多方向特征融合的室外三维目标检测方法. 计算机工程, 2023, 49 (11): 238- 246. URL
	LEI J M, YUI H, XIA Y, et al. Outdoor 3D object detection method based on multi-direction features fusion. Computer Engineering, 2023, 49 (11): 238- 246. URL
5	李朝, 兰海, 魏宪. 基于注意力的毫米波-激光雷达融合目标检测. 计算机应用, 2021, 41 (7): 2137- 2144. doi: 10.11772/j.issn.1001-9081.2020081334
	LI C, LAN H, WEI X. Attention-based object detection with millimeter wave radar-lidar fusion. Computer Applications, 2021, 41 (7): 2137- 2144. doi: 10.11772/j.issn.1001-9081.2020081334
6	WANG Y, GUIZILINI V, ZHANG T Y, et al. DETR3D: 3D object detection from multi-view images via 3D-to-2D queries[C]//Proceedings of Conference on Robot Learning. New York, USA: ACM Press, 2022: 1-10.
7	ASHISH V, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 6000-6010.
8	QIAN R, LAI X, LI X R. 3D object detection for autonomous driving: a survey. Pattern Recognition, 2022, 130, 108796. doi: 10.1016/j.patcog.2022.108796
9	WANG Y J, MAO Q Y, ZHU H Q, et al. Multi-modal 3D object detection in autonomous driving: a survey. International Journal of Computer Vision, 2023, 131, 1- 31. doi: 10.1007/s11263-022-01693-7
10	WANG C W, MA C, ZHU M, et al. PointAugmenting: cross-modal augmentation for 3D object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 11794-11803.
11	YOO J H, KIM Y, KIM J, et al. 3D-CVF: generating joint camera and lidar features using cross-view spatial feature fusion for 3D object detection[C]//Proceedings of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 720-736.
12	YIN T, ZHOU X, KRÄHENBÜHL P. Multimodal virtual point 3D detection. Advances in Neural Information Processing Systems, 2021, 34, 16494- 16507.
13	JACOBSON P, ZHOU Y, ZHAN W, et al. Center feature fusion: selective multi-sensor fusion of center-based objects[C]//Proceedings of IEEE International Conference on Robotics and Automation (ICRA). Washington D. C., USA: IEEE Press, 2023: 8312-8318.
14	LI Y, CHEN Y, QI X, et al. Unifying voxel-based representation with transformer for 3D object detection. Advances in Neural Information Processing Systems, 2022, 35, 18442- 18455. URL
15	LIU Z, TANG H, AMINI A, et al. BEVFusion: multi-task multi-sensor fusion with unified bird's-eye view representation[C]//Proceedings of IEEE International Conference on Robotics and Automation. Washington D. C., USA: IEEE Press, 2023: 2774-2781.
16	LIANG T, XIE H, YU K, et al. BEVFusion: a simple and robust LiDAR-camera fusion framework. Advances in Neural Information Processing Systems, 2022, 35, 10421- 10434.
17	ZHOU Y, TUZEL O. VoxelNet: end-to-end learning for point cloud based 3D object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 4490-4499.
18	PHILION J, FIDLER S. Lift, splat, shoot: encoding images from arbitrary camera rigs by implicitly unprojecting to 3D[C]//Proceedings of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 194-210.
19	CHEN X Y, ZHANG T Y, WANG Y, et al. FUTR3D: a unified sensor fusion framework for 3D detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2023: 172-181.
20	JIANG Q, SUN H, ZHANG X. SemanticBEVFusion: rethink LiDAR-camera fusion in unified bird's-eye view representation for 3D object detection[EB/OL]. [2023-09-20]. http://arXivpreprintarXiv:2212.04675.
21	WANG X, LEI J, LAN H, et al. DuEqNet: dual-equivariance network in outdoor 3D object detection for autonomous driving[EB/OL]. [2023-09-20]. http://arXivpreprintarXiv:2302.13577, 2023.
22	WU H, WEN C, LI W, et al. Transformation-equivariant 3D object detection for autonomous driving[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S. l.]: AAAI Press, 2023: 2795-2802.
23	CAESAR H, BANKITI V, LANG A H, et al. nuScenes: a multimodal dataset for autonomous driving[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 11621-11631.
24	COHEN T S, WELLING M. Group equivariant convolutional networks[EB/OL]. [2023-09-20]. https://arxiv.org/pdf/1602.07576.
25	LANG A H, VORA S, CAESAR H, et al. PointPillars: fast encoders for object detection from point clouds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 12697-12705.
26	CHEN Q, SUN L, WANG Z X, et al. Object as hotspots: an anchor-free 3D object detection approach via firing of hotspots[C]//Proceedings of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 68-84.
27	YIN T W, ZHOU X Y, KRAHENBUHL P. Center-based 3D object detection and tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 11784-11793.
28	YAN Y, MAO Y X, LI B. SECOND: sparsely embedded convolutional detection. Sensors, 2018, 18 (10): 3337. doi: 10.3390/s18103337
29	LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 2117-2125.
30	LIANG T T, CHU X J, LIU Y D, et al. CBNet: a composite backbone network architecture for object detection. IEEE Transactions on Image Processing, 2022, 31, 6893- 6906. doi: 10.1109/TIP.2022.3216771
31	HU J, SHEN L, SUN G, et al. Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7132-7141.
32	邹慧海, 侯进. 改进SSD算法的道路小目标检测研究. 计算机工程, 2022, 48 (5): 281- 288. URL
	ZOU H H, HOU J. Research on road small target detection with improved SSD algorithm. Computer Engineering, 2022, 48 (5): 281- 288. URL
33	BAI X Y, HU Z Y, ZHU X G, et al. Transfusion: robust LiDAR-camera fusion for 3D object detection with Transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 1090-1099.
34	GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? the KITTI vision benchmark suite[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2012: 3354-3361.
35	GEIGER A, LENZ P, STILLER C, et al. Vision meets robotics: the KITTI dataset. International Journal of Robotics Research, 2013, 32 (11): 1231- 1237. doi: 10.1177/0278364913491297
36	ZHU B J, JIANG Z K, ZHOU X X, et al. Class-balanced grouping and sampling for point cloud 3d object detection[EB/OL]. [2023-09-20]. https://arxiv.org/abs/1908.09492.

[1]	周炫余, 吴莲华, 郑勤华, 肖天星, 王紫璇, 张思敏. 联合语义提示和记忆增强的弱监督跳绳视频异常检测方法[J]. 计算机工程, 2024, 50(7): 87-95.
[2]	叶智奇, 章国宝, 朱宏伟. 一种面向室内动态行人场景的激光SLAM算法[J]. 计算机工程, 2024, 50(6): 208-217.
[3]	茹妞妞, 于晋伟, 杨卫华, 卞玮. 基于压缩与精化深度体素流模型的视频插值[J]. 计算机工程, 2022, 48(9): 248-253.
[4]	陈艳阳,曾卫明,王倪传. 基于受限玻尔兹曼机的脑功能连通性检测方法[J]. 计算机工程, 2017, 43(1): 231-236.
[5]	吴斌,孙显,王宏琦,付琨. 概率体素模型中建筑物精确三维结构提取[J]. 计算机工程, 2016, 42(10): 1-5.
[6]	冯宝,刘晓刚. 基于先验信息的脑图像数据信息提取算法[J]. 计算机工程, 2015, 41(9): 194-198.
[7]	冯宝,秦传波. 基于凸优化的脑图像数据盲信号分离算法[J]. 计算机工程, 2015, 41(8): 233-237.
[8]	冯宝,刘晓刚. 基于字典稀疏性的脑图像数据盲分离方法[J]. 计算机工程, 2015, 41(12): 269-274.
[9]	彭伟, 李建新, 闫镔, 童莉, 陈健. 基于改进空体素跳跃法的光线投射算法[J]. 计算机工程, 2012, 38(2): 264-266.
[10]	贺毅岳, 耿国华, 贾甲, 杜洪强, 贺洁琼. 基于TPS变形的3D面皮层次点对应方法[J]. 计算机工程, 2012, 38(19): 290-292.
[11]	任新宇, 范勇, 肖德成, 原飞, 张劲峰. 基于体素标号法的空间球粒体分布测量[J]. 计算机工程, 2012, 38(19): 274-276,280.
[12]	张鹏, 张建业, 王占磊, 谢文俊. 基于近邻量测认知信息的多传感器融合估计[J]. 计算机工程, 2012, 38(08): 1-3.
[13]	王庆浩;孙丰荣;王丽梅;王文明;姚桂华;张运. 一种自由臂三维超声任意切面重建方法[J]. 计算机工程, 2010, 36(9): 206-207,.

选择文件类型/文献管理软件名称

选择包含的内容