Detail-Enhanced RGB-IR Multichannel Feature Fusion Network for Semantic Segmentation

doi:10.19678/j.issn.1000-3428.0063316

Abstract

Abstract: Existing semantic segmentation methods based on deep learning are inaccurate for the edge segmentation of remote sensing images.The segmentation effect of small ground objects is poor, and the quality of RGB images seriously affects the segmentation effect.This study proposes a detail-enhanced RGB-IR multichannel feature fusion network for semantic segmentation, named MFFNet.The detail feature extraction module is used to obtain detailed features of RGB and infrared images.These are fused to generate a more distinctive feature representation and offset the missing information of RGB images relative to infrared images.While fusing the detail and high-level semantic features, the feature fusion attention module is used to adaptively generate different attention weights for each feature map to obtain an optimized feature map with more accurate semantic information and prominent detail information.The detail feature extraction and feature fusion attention module structures are designed to correspond to each other at the same level and suppress the influence of interference or irrelevant detail information when fusing high-level semantic features, highlight key detail features, and embed the channel attention module in the feature fusion attention module.This strengthens the effective fusion of high- and low-level features and generates a more discriminative feature representation, thereby improving the feature expression ability of the network.Experiments on the public Postdam dataset show that the mean intersection over union of MFFNet is 70.54%, which is an improvement of 3.95 and 4.85 percentage points compared with that of MFNet and RTFNet, respectively, and the segmentation effect for edges and small ground objects is significantly improved.

Key words: remote sensing image, deep learning, semantic segmentation, RGB-IR multichannel, detail feature extraction, feature fusion attention

摘要： 现有基于深度学习的语义分割方法对于遥感图像的地物边缘分割不准确，小地物分割效果较差，并且RGB图像质量也会严重影响分割效果。提出一种增强细节的RGB-IR多通道特征融合语义分割网络MFFNet。利用细节特征抽取模块获取RGB和红外图像的细节特征并进行融合，生成更具区分性的特征表示并弥补RGB图像相对于红外图像所缺失的信息。在融合细节特征和高层语义特征的同时，利用特征融合注意力模块自适应地为每个特征图生成不同的注意力权重，得到具有准确语义信息和突出细节信息的优化特征图。将细节特征抽取模块和特征融合注意力模块结构在同一层级上设计为相互对应，从而与高层语义特征进行融合时抑制干扰或者无关细节信息的影响，突出重要关键细节特征，并在特征融合注意力模块中嵌入通道注意力模块，进一步加强高低层特征有效融合，产生更具分辨性的特征表示，提升网络的特征表达能力。在公开的Postdam数据集上的实验结果表明，MFFNet的平均交并比为70.54%，较MFNet和RTFNet分别提升3.95和4.85个百分点，并且对于边缘和小地物的分割效果提升显著。

关键词: 遥感图像, 深度学习, 语义分割, RGB-IR多通道, 细节特征抽取, 特征融合注意力

CLC Number:

TP751

XIE Shuchun, CHEN Zhihua, SHENG Bin. Detail-Enhanced RGB-IR Multichannel Feature Fusion Network for Semantic Segmentation[J]. Computer Engineering, 2022, 48(10): 230-237,244.

谢树春, 陈志华, 盛斌. 增强细节的RGB‐IR多通道特征融合语义分割网络[J]. 计算机工程, 2022, 48(10): 230-237,244.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0063316

http://www.ecice06.com/EN/Y2022/V48/I10/230

Figures/Tables 11

References

[1] LONG J, SHELHAMER E, DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:3431-3440.
[2] BADRINARAYANAN V, KENDALL A, CIPOLLA R.SegNet:a deep convolutional encoder-decoder architecture for image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12):2481-2495.
[3] RONNEBERGER O, FISCHER P, BROX T.U-Net:convolutional networks for biomedical image segmentation[C]//Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention.Berlin, Germany:Springer, 2015:234-241.
[4] ROMERA E, ÁLVAREZ J M, BERGASA L M, et al.ERFNet:efficient residual factorized ConvNet for real-time semantic segmentation[J].IEEE Transactions on Intelligent Transportation Systems, 2018, 19(1):263-272.
[5] 谭镭, 孙怀江.SKASNet:用于语义分割的轻量级卷积神经网络[J].计算机工程, 2020, 46(9):261-267. TAN L, SUN H J.SKASNet:lightweight convolutional neural network for semantic segmentation[J].Computer Engineering, 2020, 46(9):261-267.(in Chinese)
[6] CHEN L C, ZHU Y, PAPANDREOU G, et al.Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2018:833-851.
[7] DONG R S, PAN X Q, LI F Y.DenseU-net-based semantic segmentation of small objects in urban remote sensing images[J].IEEE Access, 2019, 7:65347-65356.
[8] CUI B E, CHEN X, LU Y.Semantic segmentation of remote sensing images using transfer learning and deep convolutional neural network with dense connection[J].IEEE Access, 2020, 8:116744-116755.
[9] JIANG J, LYU C J, LIU S Y, et al.RWSNet:a semantic segmentation network based on SegNet combined with random walk for remote sensing[J].International Journal of Remote Sensing, 2020, 41(2):487-505.
[10] HU J, SHEN L, SUN G.Squeeze-and-excitation networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:7132-7141.
[11] WOO S, PARK J, LEE J Y, et al.CBAM:convolutional block attention module[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2018:3-19.
[12] WANG X L, GIRSHICK R, GUPTA A, et al.Non-local neural networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:7794-7803.
[13] 程晓悦, 赵龙章, 胡穹, 等.基于密集层和注意力机制的快速语义分割[J].计算机工程, 2020, 46(4):247-252, 259. CHENG X Y, ZHAO L Z, HU Q, et al.Fast semantic segmentation based on dense layer and attention mechanism[J].Computer Engineering, 2020, 46(4):247-252, 259.(in Chinese)
[14] GUO M Q, LIU H, XU Y Y, et al.Building extraction based on U-Net with an attention block and multiple losses[J].Remote Sensing, 2020, 12(9):1400.
[15] WANG S Q, ZHANG C, WU M.Accurate semantic segmentation in remote sensing image[C]//Proceedings of International Conference on Computing and Pattern Recognition.New York, USA:ACM Press, 2019:173-178.
[16] LIN T Y, DOLLÁR P, GIRSHICK R, et al.Feature pyramid networks for object detection[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:936-944.
[17] ZHAO H S, SHI J P, QI X J, et a1.Pyramid scene parsing network[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:2881-2890.
[18] 马震环, 高洪举, 雷涛.基于增强特征融合解码器的语义分割算法[J].计算机工程, 2020, 46(5):254-258, 266. MA Z H, GAO H J, LEI T.Semantic segmentation algorithm based on enhanced feature fusion decoder[J].Computer Engineering, 2020, 46(5):254-258, 266.(in Chinese)
[19] WANG E D, JIANG Y M, LI Y, et al.MFCSNet:multi-scale deep features fusion and cost-sensitive loss function based segmentation network for remote sensing images[J].Applied Sciences, 2019, 9(19):4043.
[20] SHANG R H, ZHANG J Y, JIAO L C, et al.Multi-scale adaptive feature fusion network for semantic segmentation in remote sensing images[J].Remote Sensing, 2020, 12(5):872.
[21] 王囡, 侯志强, 赵梦琦, 等.结合边缘检测的语义分割算法[J].计算机工程, 2021, 47(7):257-265. WANG N, HOU Z Q, ZHAO M Q, et al.Semantic segmentation algorithm combined with edge detection[J].Computer Engineering, 2021, 47(7):257-265.(in Chinese)
[22] 张娣, 陆建峰.基于双目图像与跨级特征引导的语义分割模型[J].计算机工程, 2020, 46(10):275-281, 288. ZHANG D, LU J F.Semantic segmentation model based on binocular images and guidance of cross-level features[J].Computer Engineering, 2020, 46(10):275-281, 288.(in Chinese)
[23] HAZIRBAS C, MA L, DOMOKOS C, et al.FuseNet:incorporating depth into semantic segmentation via fusion-based CNN architecture[C]//Proceedings of Asian Conference on Computer Vision.Berlin, Germany:Springer, 2017:213-228.
[24] HA Q S, WATANABE K, KARASAWA T, et al.MFNet:towards realtime semantic segmentation for autonomous vehicles with multi-spectral scenes[C]//Proceedings of International Conference on Intelligent Robots and Systems.Washington D.C., USA:IEEE Press, 2017:5108-5115.
[25] SUN Y X, ZUO W X, LIU M.RTFNet:RGB-thermal fusion network for semantic segmentation of urban scenes[J].IEEE Robotics and Automation Letters, 2019, 4(3):2576-2583.
[26] YU C Q, WANG J B, PENG C, et al.BiSeNet:bilateral segmentation network for real-time semantic segmentation[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2018:334-349.
[27] BOKHOVKIN A, BURNAEV E.Boundary loss for remote sensing imagery semantic segmentation[C]//Proceedings of International Symposium on Neural Networks.Berlin, Germany:Springer, 2019:388-401.

Please choose a citation manager

Content to export