融合RGB与IR图像的遥感小目标检测方法

doi:10.19678/j.issn.1000-3428.0069510

摘要/Abstract

摘要：

针对现有的目标检测方法在处理背景复杂、有效信息量少的遥感图像时存在的误检、漏检等问题，提出了一种多模态遥感小目标检测方法——BFMYOLO。设计了像素级的红-绿-蓝(RGB)和红外(IR)图像的融合模块, 即多模态融合模块(BFM)，充分利用不同模态的互补性，实现两种模态信息的有效融合；设计了全尺度自适应更新模块(AA)，解决特征融合过程中的多目标信息冲突问题，通过结合CARAFE上采样算子并进一步融入浅层特征，在加强非相邻层间融合的同时增强小目标的空间信息；设计了改进的任务解耦检测头(IDHead)，将分类和回归任务分开处理，以降低不同任务的相互干扰，融合深层语义特征，进一步提升模型的检测性能。采用归一化Wasserstein距离(NWD)损失函数作为定位回归损失函数，降低位置偏差的敏感性。实验结果表明，该方法在VEDAI、NWPU VHR-10和DIOR数据集上的阈值设定为0.5时的均值平均精度(mAP@0.5)分别达到78.6%、95.5%和73.3%，优于其他先进模型, 在遥感小目标检测中表现出良好的性能。

关键词: 遥感目标检测, 可见光和红外图像, 轻量级上采样算子, 注意力机制, 特征融合

Abstract:

A multimodal remote sensing small-target detection method, BFMYOLO, is proposed to address misdetection and omission issues in remote sensing images with complex backgrounds and less effective information. The method utilizes a pixel-level Red-Green-Blue (RGB) and infrared (IR) image fusion module, namely, the Bimodal Fusion Module (BFM), for effectively making full use of the complementarity of different modes to realize the effective fusion of information from two modalities. In addition, a full-scale adaptive updating module, AA, is introduced to resolve multitarget information conflicts during feature fusion. This module incorporates the CARAFE up-sampling operator and shallow features to enhance non-neighboring layer fusion and improve the spatial information of small targets. An Improved task decoupling Detection Head (IDHead) is designed to handle classification and regression tasks separately, thereby reducing the mutual interference between different tasks and enhancing detection performance by fusing deeper semantic features. The proposed method adopts the Normalized Wasserstein Distance (NWD) loss function as the localization regression loss function to mitigate positional bias sensitivity. Results of experiments on the VEDAI, NWPU VHR-10, and DIOR datasets demonstrate the superior performance of the model, with mean Average Precision when the threshold is set to 0.5 (mAP@0.5) of 78.6%, 95.5%, and 73.3%, respectively. The model thus outperforms other advanced models in remote sensing small-target detection.

Key words: remote sensing target detection, visible and infrared image, lightweight upsampling operators, attention mechanisms, feature fusion

刘春霞, 孟吉星, 潘理虎, 龚大立. 融合RGB与IR图像的遥感小目标检测方法[J]. 计算机工程, 2025, 51(7): 326-338.

LIU Chunxia, MENG Jixing, PAN Lihu, GONG Dali. Remote Sensing Small-Target Detection Method with Fusion of RGB and IR Images[J]. Computer Engineering, 2025, 51(7): 326-338.

https://www.ecice06.com/CN/Y2025/V51/I7/326

图/表 14

图1 BFMYOLO模型整体结构

Fig.1 Overall framework of BFMYOLO

图2 多模态融合模块

Fig.2 Multimodal fusion module

图3 CAFPN结构图

Fig.3 CAFPN structural diagram

图4 AA模块

Fig.4 AA module

图5 解耦检测头

Fig.5 Decoupling detection head

图6 改进的解耦检测头

Fig.6 Improved decoupling detection head

图7 主干不同块后特征融合(Fusionn表示在第n块后执行融合操作)

Fig.7 Feature fusion after different blocks in trunk (Fusionn denotes fusion operation performed after the nth block)

图8 可视化检测结果

Fig.8 Visualization detection results

参考文献 41

1	GANDHI A , ADHVARYU K , PORIA S , et al. Multimodal sentiment analysis: a systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions. Information Fusion, 2023, 91, 424- 444.
2	赫晓慧, 宋定君, 李盼乐, 等. 融合多尺度特征的遥感影像道路提取方法. 计算机工程, 2022, 48 (8): 196- 205. doi: 10.19678/j.issn.1000-3428.0062451
	HE X H , SONG D J , LI P L , et al. Remote sensing image road extraction method combined with multi-scale features. Computer Engineering, 2022, 48 (8): 196- 205. doi: 10.19678/j.issn.1000-3428.0062451
3	LU H , ZHU Y , YIN M , et al. Multimodal fusion convolutional neural network with cross-attention mechanism for internal defect detection of magnetic tile. IEEE Access, 2022, 10, 60876- 60886.
4	DU G , ZHANG L , SU K , et al. A multimodal fusion fatigue driving detection method based on heart rate and PERCLOS. IEEE Transactions on Intelligent Transportation Systems, 2022, 23 (11): 21810- 21820.
5	ROY S K , DERIA A , HONG D , et al. Multimodal fusion transformer for remote sensing image classification. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61, 5515620.
6	赵恩源, 宋宁, 聂婕, 等. 面向遥感视觉问答的尺度引导融合推理网络. 软件学报, 2024, 35 (5): 2133- 2149.
	ZHAO E Y , SONG N , NIE J , et al. Scale-guided fusion inference network for remote sensing visual question answering. Journal of Software, 2024, 35 (5): 2133- 2149.
7	付琨, 王佩瑾, 冯瑛超, 等. 遥感跨模态智能解译: 模型、数据与应用. 中国科学: 信息科学, 2023, 53 (8): 1529- 1559.
	FU K , WANG P J , FENG Y C , et al. Cross-modal remote sensing intelligent interpretation: method, data, and applications. Chinese Science Informationis, 2023, 53 (8): 1529- 1559.
8	孙汉淇, 潘晨, 何灵敏, 等. 多模态特征融合的遥感图像语义分割网络. 计算机工程与应用, 2022, 58 (24): 256- 264.
	SUN H Q , PAN C , HE L M , et al. Remote sensing image semantic segmentation network based on multimodal feature fusion. Computer Engineering and Applications, 2022, 58 (24): 256- 264.
9	SUN Y , FU Z , SUN C , et al. Deep multimodal fusion network for semantic segmentation using remote sensing image and LiDAR data. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60, 5404418.
10	LIN T, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2017: 936-944.
11	REDMON J, FARHADI A. Yolov3: an incremental improvement[EB/OL]. (2018-04-08)[2024-03-20]. https://arxiv.org/abs/1804.02767.
12	LIN T, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C]//Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV). Washington D. C., USA: IEEE Press, 2017: 2999-3007.
13	KONG T, SUN F, HUANG W, et al. Deep feature pyramid reconfiguration for object detection[C]//Proceedings of Computer Vision-ECCV 2018. Berlin, Germany: Springer, 2018: 172-188.
14	LIU S, QI L, QIN H, et al. Path aggregation network for instance segmentation[EB/OL]. (2018-09-18)[2024-03-20]. https://arxiv.org/abs/1803.01534.
15	GUO C, FAN B, ZHANG Q, et al. AugFPN: improving multi-scale feature learning for object detection[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2020: 12592-12601.
16	CHEN S , ZHAO J , ZHOU Y , et al. Info-FPN: An Informative Feature Pyramid Network for object detection in remote sensing images. Expert Systems with Applications, 2023, 214, 119132.
17	LI Z , LI E , XU T , et al. Feature alignment FPN for oriented object detection in remote sensing images. IEEE Geoscience and Remote Sensing Letters, 2023, 20, 6001705.
18	DONG X , QIN Y , GAO Y , et al. Attention-based multi-level feature fusion for object detection in remote sensing images. Remote Sensing, 2022, 14 (15): 3735.
19	WANG J W, XU C, YANG W, et al. A normalized gaussian Wasserstein distance for tiny object detection[EB/OL]. (2022-06-14)[2024-03-20]. https://arxiv.org/abs/2110.13389.
20	ZHANG J , LEI J , XIE W , et al. SuperYOLO: super resolution assisted object detection in multimodal remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61, 5605415.
21	YANG G, FENG W, JIN J, et al. Face mask recognition system with YOLOV5 based on image recognition[C]//Proceedings of 2020 IEEE 6th International Conference on Computer and Communications (ICCC). Washington D. C., USA: IEEE Press, 2020: 1398-1404.
22	WANG J, CHEN K, XU R, et al. CARAFE: content-aware reassembly of features[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 3007-3016.
23	RAZAKARIVONY S , JURIE F . Vehicle detection in aerial imagery : a small target detection benchmark. Journal of Visual Communication and Image Representation, 2016, 34, 187- 203.
24	PANG J , CHEN K , LI Q , et al. Towards balanced learning for instance recognition. International Journal of Computer Vision, 2021, 129 (5): 1376- 1393.
25	LIU Y , LI Q , YUAN Y , et al. ABNet: adaptive balanced network for multiscale object detection in remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60, 3133956.
26	BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: optimal speed and accuracy of object detection[EB/OL]. (2020-04-23)[2024-03-20]. https://arxiv.org/abs/2004.10934.
27	SHARMA M , DHANARAJ M , KARNAM S , et al. YOLOrs: object detection in multimodal remote sensing imagery. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14, 1497- 1508.
28	PHAM M T , COURTRAI L , FRIGUET C , et al. YOLO-fine: one-stage detector of small objects under various backgrounds in remote sensing Images. Remote Sensing, 2020, 12 (15): 2501.
29	FANG Q , WANG Z . Cross-modality attentive feature fusion for object detection in multispectral remote sensing imagery. Pattern Recognition, 2022, 130, 108786.
30	NIE J , SUN H , SUN X , et al. Cross-modal feature fusion and interaction strategy for CNN-transformer-based object detection in visual and infrared remote sensing imagery. IEEE Geoscience and Remote Sensing Letters, 2024, 21, 5000405.
31	SHEN J , CHEN Y , LIU Y , et al. ICAFusion: iterative cross-attention guided feature fusion for multispectral object detection. Pattern Recognition, 2024, 145, 109913.
32	LI K , WAN G , CHENG G , et al. Object detection in optical remote sensing images: a survey and a new benchmark. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 159, 296- 307.
33	CHENG G , HAN J , ZHOU P , et al. Multi-class geospatial object detection and geographic image classification based on collection of part detectors. ISPRS Journal of Photogrammetry and Remote Sensing, 2014, 98, 119- 132.
34	ZHANG S, CHI C, YAO Y, et al. Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2020: 9756-9765.
35	HUANG X, WANG X, LV W, et al. PP-YOLOv2: a practical object detector[EB/OL]. (2021-04-21)[2024-03-20]. https://arxiv.org/abs/2104.10419.
36	LI X, WANG W, WU L, et al. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection[EB/OL]. (2020-06-08)[2024-03-20]. https://arxiv.org/abs/2006.04388.
37	REN S , HE K , GIRSHICK R , et al. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2017, 39 (6): 1137- 1149.
38	SANDLER M, HOWARD A, ZHU M, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 4510-4520.
39	ZHANG X, ZHOU X, LIN M, et al. ShuffleNet: an extremely efficient convolutional neural network for mobile devices[EB/OL]. (2017-07-04)[2024-03-20]. https://arxiv.org/abs/1707.01083v2.
40	YANG Y , SUN X , DIAO W , et al. Adaptive knowledge distillation for lightweight remote sensing object detectors optimizing. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60, 5623715.
41	LI J , ZHANG Z , TIAN Y , et al. Target-guided feature super-resolution for vehicle detection in remote sensing images. IEEE Geoscience and Remote Sensing Letters, 2022, 19, 8020805.

[1]	沙宇洋, 陆京涛, 杜浩凡, 翟小兵, 孟维宇, 廉旭, 罗刚, 李克峰. 适用于导盲场景的多尺度特征融合轻量化道路图像分割算法[J]. 计算机工程, 2025, 51(7): 314-325.
[2]	栾孟娜, 郑秋梅, 王风华. 基于DMC-YOLO的交通标志实时检测算法[J]. 计算机工程, 2025, 51(7): 90-99.
[3]	周莎, 车生兵, 考友琛, 张旭, 郭甚驿. 基于特征选择和时空特征的网络入侵检测[J]. 计算机工程, 2025, 51(7): 223-231.
[4]	宋杰, 徐慧英, 朱信忠, 黄晓, 陈晨, 王泽宇. 基于YOLOv8改进的跌倒检测算法: OEF-YOLO[J]. 计算机工程, 2025, 51(7): 127-139.
[5]	彭菊红, 张弛, 高谦, 张光明, 谈栋华, 赵明俊. 基于改进的YOLOv8算法的钢材缺陷检测[J]. 计算机工程, 2025, 51(7): 152-160.
[6]	曹蓓, 赵奎. 基于双重情感和多特征融合的虚假新闻检测[J]. 计算机工程, 2025, 51(6): 193-203.
[7]	赵小虎, 谢礼逊, 慕灯聪, 张悦. 基于TCM-YOLO网络的金属表面缺陷检测方法[J]. 计算机工程, 2025, 51(6): 338-348.
[8]	李毅, 徐慧英, 朱信忠, 黄晓, 王舒梦, 李悉钰. 基于YOLOv5n模型改进的口罩检测算法: Mask-YOLO[J]. 计算机工程, 2025, 51(6): 297-310.
[9]	华家宝, 张京瑞, 朱福民, 陈璐. 基于路侧相机的自适应空间变换车辆检测方法[J]. 计算机工程, 2025, 51(6): 349-359.
[10]	刘凯, 任洪逸, 李蓥, 季怡, 刘纯平. 基于交叉模态注意力特征增强的医学视觉问答[J]. 计算机工程, 2025, 51(6): 49-56.
[11]	李白芽. 基于CNN-Transformer的电子喉镜病灶及器官分割网络[J]. 计算机工程, 2025, 51(6): 327-337.
[12]	单鹏畅, 高利剑, 董文龙, 毛启容. 基于显著目标追踪的行为检测方法[J]. 计算机工程, 2025, 51(6): 93-101.
[13]	郑诚, 李鹏飞. 基于双超图神经网络特征融合的文本分类[J]. 计算机工程, 2025, 51(6): 127-135.
[14]	马月坤, 马铭佑. 基于全局与局部特征加权融合的隐喻识别模型[J]. 计算机工程, 2025, 51(5): 143-153.
[15]	解庆, 张凌峰, 马艳春, 刘永坚. 基于反射分类与梯度恢复的单幅图像去反射模型[J]. 计算机工程, 2025, 51(4): 227-238.

选择文件类型/文献管理软件名称

选择包含的内容