3D Object Detection Algorithm Based on 4D Millimeter-Wave Radar and Vision Fusion

doi:10.19678/j.issn.1000-3428.0070113

Abstract

Abstract:

This study proposes a Common and Differential Cross-Attention Module-Bird's-Eye View (CDCAM-BEV) algorithm that combines 4D millimeter-wave radar and vision fusion to improve target detection accuracy for pedestrian and vehicle target recognition and localization in autonomous driving scenarios. First, a radar cylinder network is designed to encode the 4D radar point cloud into a pseudo image and convert the monocular image into a Bird's-Eye View (BEV) feature through Orthogonal Feature Transformation (OFT). Second, based on the cross-attention mechanism, a Common Information Extraction Module (CICAM) and a Differential Information Extraction Module (DICAM) are used to fully explore the common and differential information between radar and images. Finally, a BEV feature fusion module is designed based on CICAM and DICAM to achieve feature-level fusion of image and radar information in the BEV space. Experiments are conducted on the VOD dataset, and the CDCAM-BEV algorithm is compared with five other 3D object detection algorithms. The experimental results show that CDCAM-BEV achieves better detection performance in multiple modes. In the 3D mode, the average detection accuracy of CDCAM-BEV is 3.65 percentage points higher than that of the second ranked Part-A²; in the BEV mode, it is 5.04 percentage points higher than that of the second ranked PointPillars; in the Average Directional Similarity (AOS) mode, it is 2.62 percentage points higher than that of the second ranked Part-A². These results show that CDCAM-BEV exhibits excellent performance in all modes, effectively fusing images and 4D radar point cloud features, which significantly improves the accuracy and reliability of object detection.

Key words: 4D millimeter-wave radar, Bird's-Eye View (BEV), autonomous driving, cross attention mechanism, 3D object detection

摘要：

针对自动驾驶场景中行人和车辆的目标识别与定位问题, 提出一种四维(4D)毫米波雷达与视觉融合的CDCAM-BEV算法, 以提高目标检测的精度。首先, 设计雷达柱体网络, 将4D雷达点云编码为伪图像, 并通过正交特征变换(OFT)将单目图像转换为鸟瞰图(BEV)特征; 其次, 基于交叉注意力机制, 设计共同信息提取模块(CICAM)和差异信息提取模块(DICAM), 充分挖掘雷达和图像的公共信息和差异信息; 最后, 基于CICAM和DICAM设计BEV特征融合模块, 实现图像信息和雷达信息在BEV空间的特征级融合。在VOD数据集上进行实验, 将CDCAM-BEV算法与其他5种三维(3D)目标检测算法进行对比。实验结果表明, CDCAM-BEV在多个模式下检测性能均优于其他算法。在3D模式下, CDCAM-BEV的平均检测精度比排名第二的Part-A²高出3.65百分点; 在BEV模式下, 比排名第二的PointPillars高出5.04百分点; 在平均方向相似度(AOS)模式下, 比排名第二的Part-A²高出2.62百分点。结果显示, CDCAM-BEV在各模式下均表现出卓越性能, 能够有效融合图像和4D雷达点云特征, 显著提高目标检测的精度和可靠性。

关键词: 四维毫米波雷达, 鸟瞰图, 自动驾驶, 交叉注意力机制, 三维目标检测

LI Jianlang, WU Xindian, CHEN Ling, YANG Bo, TANG Wensheng. 3D Object Detection Algorithm Based on 4D Millimeter-Wave Radar and Vision Fusion[J]. Computer Engineering, 2026, 52(2): 299-310.

李健浪, 吴新电, 陈灵, 阳波, 唐文胜. 基于4D毫米波雷达与视觉融合的三维目标检测算法[J]. 计算机工程, 2026, 52(2): 299-310.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0070113

https://www.ecice06.com/EN/Y2026/V52/I2/299

Figures/Tables 15

Fig.1 Overall structure of CDCAM-BEV

Fig.2 The process of generating orthogonal feature maps based on OFT

Fig.3 Image BEV feature coding module

Fig.4 4RPFNet structure

Fig.5 The structure of BEV feature fusion module

Fig.6 The structure of CICAM

Fig.7 The structure of DICAM

Fig.8 Experimental results of CDCAM-BEV on VOD dataset

Fig.9 Comparison of different fusion strategies

References 32

1	王若萱, 吴建平, 徐辉. 自动驾驶汽车感知系统仿真的研究及应用综述. 系统仿真学报, 2022, 34 (12): 2507- 2521.
	WANG R X , WU J P , XU H . Overview of research and application on autonomous vehicle oriented perception system simulation. Journal of System Simulation, 2022, 34 (12): 2507- 2521.
2	AYALA R , MOHD T K . Sensors in autonomous vehicles: a survey. Journal of Autonomous Vehicles and Systems, 2021, 1 (3): 031003. doi: 10.1115/1.4052991
3	TAN R T. Visibility in bad weather from a single image[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2008: 1-8.
4	陈熙源, 戈明明, 姚志婷, 等. 雨雪天气下的激光雷达滤波算法研究. 仪器仪表学报, 2023, 44 (7): 172- 181.
	CHEN X Y , GE M M , YAO Z T , et al. Research on lidar filtering algorithm for rainy and snowy weather. Chinese Journal of Scientific Instrument, 2023, 44 (7): 172- 181.
5	SUN R , SUZUKI K , OWADA Y , et al. A millimeter-wave automotive radar with high angular resolution for identification of closely spaced on-road obstacles. Scientific Reports, 2023, 13, 3233. doi: 10.1038/s41598-023-30406-4
6	任珈民, 宫宁生, 韩镇阳. 基于YOLOv3与卡尔曼滤波的多目标跟踪算法. 计算机应用与软件, 2020, 37 (5): 169- 176.
	REN J M , GONG N S , HAN Z Y . Multi-target tracking algorithm based on YOLOv3 and Kalman filter. Computer Applications and Software, 2020, 37 (5): 169- 176.
7	BAI J , LI S , HUANG L B , et al. Robust detection and tracking method for moving object based on radar and camera data fusion. IEEE Sensors Journal, 2021, 21 (9): 10761- 10774. doi: 10.1109/JSEN.2021.3049449
8	BANSAL K, RUNGTA K, BHARADIA D. RadSegNet: a reliable approach to radar camera fusion[EB/OL]. [2024-05-05]. https://arxiv.org/abs/2208.03849.
9	NABATI R, QI H R. CenterFusion: center-based radar and camera fusion for 3D object detection[C]//Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV). Washington D.C., USA: IEEE Press, 2021: 1526-1535.
10	TAN B , MA Z X , ZHU X C , et al. 3-D object detection for multiframe 4-D automotive millimeter-wave radar point cloud. IEEE Sensors Journal, 2023, 23 (11): 11125- 11138. doi: 10.1109/JSEN.2022.3219643
11	BAI J , LI S , TAN B , et al. Traffic participants classification based on 3D radio detection and ranging point clouds. IET Radar, Sonar & Navigation, 2022, 16 (2): 278- 290.
12	MEYER M, KUSCHK G. Automotive radar dataset for deep learning based 3D object detection[C]//Proceedings of the 16th European Radar Conference (EuRAD). Washington D.C., USA: IEEE Press, 2019: 129-132.
13	PALFFY A , POOL E , BARATAM S , et al. Multi-class road user detection with 3 1D radar in the view-of-delft dataset. IEEE Robotics and Automation Letters, 2022, 7 (2): 4961- 4968. doi: 10.1109/LRA.2022.3147324
14	PAEK D H, KONG S H, WIJAYA K T. K-Radar: 4D radar object detection for autonomous driving in various weather conditions[EB/OL]. [2024-05-05]. https://arxiv.org/abs/2206.08171.
15	ZHENG L Q, MA Z X, ZHU X C, et al. TJ4DRadSet: a 4D radar dataset for autonomous driving[C]//Proceedings of the 25th IEEE International Conference on Intelligent Transportation Systems (ITSC). Washington D.C., USA: IEEE Press, 2022: 493-498.
16	LIU Z J, TANG H T, AMINI A, et al. BEVFusion: multi-task multi-sensor fusion with unified bird's-eye view representation[C]//Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Washington D.C., USA: IEEE Press, 2023: 2774-2781.
17	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2024-05-05]. https://arxiv.org/abs/1409.1556.
18	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2016: 770-778.
19	REDMON J, FARHADI A. YOLOv3: an incremental improvement[EB/OL]. [2024-05-05]. https://arxiv.org/abs/1804.02767.
20	CHADWICK S, MADDERN W, NEWMAN P. Distant vehicle detection using radar and vision[C]//Proceedings of the International Conference on Robotics and Automation (ICRA). Washington D.C., USA: IEEE Press, 2019: 8311-8317.
21	LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot MultiBox detector[EB/OL]. [2024-05-05]. https://arxiv.org/abs/1512.02325.
22	CHANG S , ZHANG Y F , ZHANG F , et al. Spatial attention fusion for obstacle detection using MmWave radar and vision sensor. Sensors, 2020, 20 (4): 956. doi: 10.3390/s20040956
23	KIM Y, CHOI J W, KUM D. GRIF Net: gated region of interest fusion network for robust 3D object detection from radar point cloud and monocular image[C]//Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Washington D.C., USA: IEEE Press, 2021: 10857-10864.
24	LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2017: 936-944.
25	REN M Y, POKROVSKY A, YANG B, et al. SBNet: sparse blocks network for fast inference[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2018: 8711-8720.
26	KIM Y , KIM S , CHOI J W , et al. CRAFT: camera-radar 3D object detection with spatio-contextual fusion transformer. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37 (1): 1160- 1168. doi: 10.1609/aaai.v37i1.25198
27	WU Z Z, CHEN G L, GAN Y Z, et al. MVFusion: multi-view 3D object detection with semantic-aligned radar and camera fusion[C]//Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). Washington D.C., USA: IEEE Press, 2023: 2766-2773.
28	ZHENG L Q , LI S , TAN B , et al. RCFusion: fusing 4-D radar and camera with bird's-eye view features for 3-D object detection. IEEE Transactions on Instrumentation and Measurement, 2023, 72, 8503814.
29	RODDICK T, KENDALL A, CIPOLLA R. Orthographic feature transform for monocular 3D object detection[EB/OL]. [2024-05-05]. https://arxiv.org/abs/1811.08188.
30	LANG A H, VORA S, CAESAR H, et al. PointPillars: fast encoders for object detection from point clouds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2020: 12689-12697.
31	YAN H, XIONG S L, WANG L, et al. ATFusion: an alternate cross-attention transformer network for infrared and visible image fusion[EB/OL].[2024-05-05]. https://arxiv.org/abs/2401.11675.
32	YAN Y , MAO Y X , LI B . SECOND: sparsely embedded convolutional detection. Sensors, 2018, 18 (10): 3337. doi: 10.3390/s18103337

[1]	ZHANG Zhaoxin, HUANG Shize, ZHANG Bingjie, SHEN Tuo. Camouflaged Adversarial Example Generation Method for the Form of Motion Blur in Traffic Scenes [J]. Computer Engineering, 2025, 51(3): 45-53.
[2]	YAN Fuwu, WANG Zhanbin, HU Jie, XU Wencai, TANG Yuxuan, CHEN Nan. Research on Structured Road Vehicle Trajectory Prediction Method with Recognizable Maneuver Intentions [J]. Computer Engineering, 2025, 51(11): 355-365.
[3]	LIU Yanghong, FU Yangyouran, DONG Xingping. HDMapFusion: High-Definition Map Generation with Multi-Modality Fusion for Autonomous Driving(Invited) [J]. Computer Engineering, 2025, 51(10): 18-26.
[4]	Ci XIAO, Yang XU, Yongdan ZHANG, Mingwen FENG, Yiqian HUANG. Nighttime Semantic Segmentation with Attention and Low-Light Enhancement [J]. Computer Engineering, 2024, 50(7): 271-281.
[5]	FU Mingjian, GUO Fuqiang. Research on Decision-Making at Intersection Without Traffic Lights Based on Deep Reinforcement Learning [J]. Computer Engineering, 2024, 50(5): 91-99.
[6]	LIU Hongwei, SHAO Dongheng, YANG Jian, WEI Xian, LI Ke, YOU Xiong. Multi-Level Rotational Equivariant Object Detection Network Based on BEV Fusion [J]. Computer Engineering, 2024, 50(11): 246-257.
[7]	Rong FEI, Mengyang MA, Xiao ZHANG, Xinhong HEI, Qingzheng XU, Yuan QIU. Collision Detection Model for Autonomous Driving Based on Trajectory Prediction and Conflict Detection [J]. Computer Engineering, 2023, 49(7): 10-20.
[8]	LI Xuesong, ZHANG Qieshi, SONG Chengqun, KANG Yuhang, CHENG Jun. Review of Trajectory Prediction Technology in Autonomous Driving Scenes [J]. Computer Engineering, 2023, 49(5): 1-11.
[9]	Jiaming LEI, Hui YU, Yu XIA, Jielong GUO, Xian WEI. Outdoor 3D Object Detection Method Based on Multi-Direction Features Fusion [J]. Computer Engineering, 2023, 49(11): 238-246.
[10]	DU Tian, LI Xin, LAI Chengzhe, ZHENG Dong. Secure Trust Management Scheme for Autonomous Driving Map Updating [J]. Computer Engineering, 2022, 48(6): 154-166.
[11]	SONG Jiayan, SU Shengchao. Multi-Vehicle Collaborative Motion Planning for Autonomous Driving Based on Improved Ant Colony Optimization Algorithm [J]. Computer Engineering, 2022, 48(11): 299-305,313.
[12]	ZHANG Yongliang, LU Yang, ZHU Wuqiang, WEI Xing, WEI Zhen. Traffic Sign Detection Based on Multi-Scale Feature Extraction and Feature Fusion [J]. Computer Engineering, 2022, 48(10): 270-278,287.
[13]	DUAN Jianmin,TIAN Xiaosheng,XIA Tian,HUA Xiaofeng. Trapezoidal Speed Planning Method of Intelligent Vehicle Based on Intermediate Speed [J]. Computer Engineering, 2018, 44(8): 301-307,314.
[14]	QIAO Liang,BAO Hong,XUAN Zuxing,LIANG Jun,PAN Feng. Autonomous Driving Ramp Merging Model Based on Reinforcement Learning [J]. Computer Engineering, 2018, 44(7): 20-24,31.
[15]	GAO Xiang,LI Zhihao,YANG Hui. Unmanned Autonomous Driving Method Based on Content-centric Network [J]. Computer Engineering, 2015, 41(12): 305-311.

Please choose a citation manager

Content to export