Real-Time Semantic Segmentation Algorithm Based on Multi-Scale Feature Fusion and Polarized Self-Attention Mechanism

doi:10.19678/j.issn.1000-3428.0070747

Abstract

Abstract:

Real-time semantic segmentation, one of the core tasks in computer vision, plays a crucial role in various applications such as autonomous driving and traffic control. Existing real-time semantic segmentation algorithms based on the encoder—decoder structure often sacrifice segmentation accuracy to achieve real-time performance. However, to ensure real-time performance, these algorithms typically have a small receptive field, resulting in poor segmentation performance for large-scale objects in road scenes. To address this issue, this paper proposes a real-time semantic segmentation algorithm tailored for road scenes based on the encoder—decoder structure. First, a Multi-scale Feature Fusion (MFF) mechanism is designed in the feature extraction stage to effectively fuse receptive field features on a larger scale, thereby enhancing the segmentation performance for large-scale objects. Second, a Polarized Self-Attention (PSA) mechanism is incorporated at the end of the encoder to enhance the local perception within the large-scale receptive field, further improving the segmentation performance for large-scale objects. Experimental results on the Cityscapes and Camvid datasets show that, when using a single NVIDIA RTX 3090 GPU, the algorithm achieves Mean Intersection over Union (MIoU) scores of 80.6% and 81.1% at frame rates of 43.5 frame/s and 91.2 frame/s, respectively. These results show that the proposed algorithm achieves higher segmentation accuracy than the comparative algorithm.

Key words: semantic segmentation, encoder—decoder structure, Multi-scale Feature Fusion (MFF), self-attention mechanism, large-scale objects

摘要：

实时语义分割作为计算机视觉领域的核心任务之一, 在无人驾驶、交通管控等诸多方面均发挥着极为关键的作用。现有基于编码器-解码器结构的实时语义分割算法通常通过牺牲分割精度来达到实时效果, 然而, 这类算法为了保证实时性, 其感受野通常较小, 从而导致对道路场景中的大尺度物体分割效果较差。为此, 基于编码器-解码器结构, 提出一种针对道路场景的实时语义分割算法。首先, 在特征提取阶段设计一个多尺度特征融合(MFF)机制, 对较大尺度内的感受野特征进行有效融合, 提升对大尺度物体的分割效果; 然后, 在编码器末端融入一个极化自注意力(PSA)机制, 增强大尺度感受野中的局部感知, 进一步提升对大尺度物体的分割效果。在数据集Cityscapes与Camvid上进行测试, 实验结果表明, 采用单个NVIDIA RTX 3090 GPU时, 该算法在43.5帧/s和91.2帧/s的帧率下分别取得了80.6%和81.1%的平均交并比(MIoU), 相较对比算法获得了更高的分割精度。

关键词: 语义分割, 编码器-解码器结构, 多尺度特征融合, 自注意力机制, 大尺度物体

LIU Xiangbin, FANG Cheng, LIU Shuai. Real-Time Semantic Segmentation Algorithm Based on Multi-Scale Feature Fusion and Polarized Self-Attention Mechanism[J]. Computer Engineering, 2026, 52(7): 210-219.

刘相滨, 方诚, 刘帅. 基于多尺度特征融合与极化自注意力机制的实时语义分割算法[J]. 计算机工程, 2026, 52(7): 210-219.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0070747

https://www.ecice06.com/EN/Y2026/V52/I7/210

Figures/Tables 12

Fig.1 Framework of real-time semantic segmentation algorithm

Fig.2 Structure of MFF-CSP

Fig.3 Multi-scale feature fusion mechanism

Fig.4 Polarized self-attention mechanism

Fig.5 Visualization results of the baseline algorithm and MFFPSA algorithm on the Cityscapes validation set

Fig.6 Visualization results of different variant methods on the Cityscapes test set

References 29

1	闵莉, 董冰洁, 安冬. 基于多注意力机制与跨特征融合的语义分割算法. 计算机工程, 2024, 50 (8): 282- 289. doi: 10.19678/j.issn.1000-3428.0067682
	MIN L , DONG B J , AN D . Semantic segmentation algorithm based on multi-attention mechanism and cross-feature fusion. Computer Engineering, 2024, 50 (8): 282- 289. doi: 10.19678/j.issn.1000-3428.0067682
2	YE Z J , WEI Y , ZHANG W D , et al. An efficient real-time vehicle monitoring method. IEEE Transactions on Intelligent Transportation Systems, 2022, 23 (11): 22073- 22083. doi: 10.1109/TITS.2022.3150224
3	高常鑫, 徐正泽, 吴东岳, 等. 深度学习实时语义分割综述. 中国图象图形学报, 2024, 29 (5): 1119- 1145.
	GAO C X , XU Z Z , WU D Y , et al. Deep learning-based real-time semantic segmentation: a survey. Journal of Image and Graphics, 2024, 29 (5): 1119- 1145.
4	YANG S , ZHANG L , LIU S , et al. Real-time semantic segmentation via a densely aggregated bilateral network. IEEE Transactions on Neural Networks and Learning Systems, 2025, 36 (1): 381- 392. doi: 10.1109/TNNLS.2023.3326665
5	WU D Y , GUO Z L , LI A Y , et al. Conditional boundary loss for semantic segmentation. IEEE Transactions on Image Processing, 2023, 32, 3717- 3731. doi: 10.1109/TIP.2023.3290519
6	LIANG Z Y , DAI X D , WU Y Q , et al. Multi-granularity context network for efficient video semantic segmentation. IEEE Transactions on Image Processing, 2023, 32, 3163- 3175. doi: 10.1109/TIP.2023.3269982
7	FAN J Q , WANG F , CHU H Q , et al. MLFNet: multi-level fusion network for real-time semantic segmentation of autonomous driving. IEEE Transactions on Intelligent Vehicles, 2023, 8 (1): 756- 767. doi: 10.1109/TIV.2022.3176860
8	SHI X Q , YIN Z Y , HAN G J , et al. BSSNet: a real-time semantic segmentation network for road scenes inspired from AutoEncoder. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34 (5): 3424- 3438. doi: 10.1109/TCSVT.2023.3325360
9	CHEN L , DAI H H , ZHENG Y . RAFNet: reparameterizable across-resolution fusion network for real-time image semantic segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34 (2): 1212- 1227. doi: 10.1109/TCSVT.2023.3293166
10	PASZKE A, CHAURASIA A, KIM S, et al. ENet: a deep neural network architecture for real-time semantic segmentation[EB/OL]. [2024-10-05]. https://arxiv.org/abs/1606.02147.
11	NIRKIN Y, WOLF L, HASSNER T. HyperSeg: patch-wise hypernetwork for real-time semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2021: 4060-4069.
12	PAN H H , HONG Y D , SUN W C , et al. Deep dual-resolution networks for real-time and accurate semantic segmentation of traffic scenes. IEEE Transactions on Intelligent Transportation Systems, 2023, 24 (3): 3448- 3460. doi: 10.1109/TITS.2022.3228042
13	LI K G , GENG Q C , WAN M X , et al. Context and spatial feature calibration for real-time semantic segmentation. IEEE Transactions on Image Processing, 2023, 32, 5465- 5477. doi: 10.1109/TIP.2023.3318967
14	XU J C, XIONG Z X, BHATTACHARYYA S P. PIDNet: a real-time semantic segmentation network inspired by PID controllers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 19529-19539.
15	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[EB/OL]. [2024-10-05]. https://arxiv.org/abs/1505.04597.
16	BADRINARAYANAN V , KENDALL A , CIPOLLA R . SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (12): 2481- 2495. doi: 10.1109/TPAMI.2016.2644615
17	CHEN L C , PAPANDREOU G , KOKKINOS I , et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40 (4): 834- 848. doi: 10.1109/TPAMI.2017.2699184
18	WASWANI A, SHAZEER N, PARMAR J, et al. Attention is all you need[EB/OL]. [2024-10-05]. https://arxiv.org/abs/1706.03762.
19	HE K M, ZHANG X Y, REN S Q, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[EB/OL]. [2024-10-05]. https://arxiv.org/abs/1406.4729.
20	CORDTS M, OMRAN M, RAMOS S, et al. The Cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2016: 3213-3223.
21	BROSTOW G J , FAUQUEUR J , CIPOLLA R . Semantic object classes in video: a high-definition ground truth database. Pattern Recognition Letters, 2009, 30 (2): 88- 97. doi: 10.1016/j.patrec.2008.04.005
22	FAN M Y, LAI S Q, HUANG J S, et al. Rethinking BiSeNet for real-time semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2021: 9711-9720.
23	CHEN L W, GU L, ZHENG D Z, et al. Frequency-adaptive dilated convolution for semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2024: 3414-3425.
24	WAN Q, HUANG Z L, LU J C, et al. SeaFormer++: squeeze-enhanced axial Transformer for mobile visual recognition[EB/OL]. [2024-10-05]. https://arxiv.org/abs/2301.13156.
25	ZHOU L G , CHEN G , LIU L , et al. Real-time semantic segmentation in traffic scene using cross stage partial-based encoder-decoder network. Engineering Applications of Artificial Intelligence, 2023, 126, 106901. doi: 10.1016/j.engappai.2023.106901
26	王卓, 瞿绍军. 基于注意力机制与多尺度池化的实时语义分割网络. 计算机工程, 2023, 49 (10): 222-229, 238. doi: 10.19678/j.issn.1000-3428.0065885
	WANG Z , QU S J . Real-time semantic segmentation network based on attention mechanism and multi-scale pooling. Computer Engineering, 2023, 49 (10): 222-229, 238. doi: 10.19678/j.issn.1000-3428.0065885
27	HE J Y , LIANG S H , WU X , et al. MGSeg: multiple granularity-based real-time semantic segmentation network. IEEE Transactions on Image Processing, 2021, 30, 7200- 7214. doi: 10.1109/TIP.2021.3102509
28	YU C Q , GAO C X , WANG J B , et al. BiSeNetV2: bilateral network with guided aggregation for real-time semantic segmentation. International Journal of Computer Vision, 2021, 129 (11): 3051- 3068. doi: 10.1007/s11263-021-01515-2
29	郑秋梅, 牛薇薇, 王风华, 等. 基于细节增强的双分支实时语义分割网络. 计算机应用, 2024, 44 (10): 3058- 3066.
	ZHENG Q M , NIU W W , WANG F H , et al. Dual-branch real-time semantic segmentation network based on detail enhancement. Journal of Computer Applications, 2024, 44 (10): 3058- 3066.

[1]	HU Kangyuan, GUO Tao, MU Nan. Image Inpainting Method for Cultural Relics Based on Self-Attention Mechanism and Dynamic Masking Mechanism [J]. Computer Engineering, 2026, 52(6): 179-188.
[2]	SONG Tianze, CAO Congjun, HE Jiaqi, WANG Xusheng, LIU Chenyu. Research on Dense Pedestrian Detection Algorithm Based on Improved DETR [J]. Computer Engineering, 2026, 52(5): 250-258.
[3]	SU Jianhua, CHI Yunxian, XU Yunfeng, GAO Kai. Multimodal Intent Recognition Based on Attention Modality Fusion [J]. Computer Engineering, 2026, 52(3): 234-242.
[4]	LIU Xiaoyu, LIAO Zhifang, TAN Sui, YU Zhiwu. Bridge Dynamic Strain Prediction Based on Stacked GRU Neural Network [J]. Computer Engineering, 2026, 52(3): 441-450.
[5]	LI Dongfeng, CHEN Yuren, YU Bo. Pavement Crack Detection Method Based on Multi-Level Feature Fusion [J]. Computer Engineering, 2026, 52(1): 154-165.
[6]	LI Bingye, BAO Yu, CAO Wei, LI Mingze, XU Fangzheng. Cross-layer Channel Pruning Method Based on Variable Sequences [J]. Computer Engineering, 2026, 52(1): 381-389.
[7]	ZHAI Zhipeng, CAO Yang, SHEN Qinqin, SHI Quan. Traffic Flow Prediction Based on Multiple Spatio-Temporal Graph Fusion and Dynamic Attention [J]. Computer Engineering, 2025, 51(9): 139-148.
[8]	LIU Yunxiang, LIANG Zhichao. A Highly Efficient Traffic Prediction Model for Continuous Time-series Graph Attention Networks [J]. Computer Engineering, 2025, 51(4): 350-359.
[9]	YANG Ping, ZHANG Xi. Improved DeepLabv3+ Road Surface Crack Detection Method [J]. Computer Engineering, 2025, 51(4): 261-270.
[10]	SUN Ting, YANG Jie, LI Jiaxuan, WANG Yaozong. Optimization of YOLOv7 Road Sign Detection Algorithm for Low-Light Traffic Scenes [J]. Computer Engineering, 2025, 51(3): 342-351.
[11]	LIU Zhoufeng, LI Bingrui, YANG Ruimin, LI Chunlei, HE Yuan, DING Shumin. Research on Weakly Supervised Semantic Segmentation Algorithm Based on Modulation-Global Reasoning [J]. Computer Engineering, 2025, 51(2): 344-355.
[12]	LEI Saiyue, FANG Li, LI Chende, YANG Ming. Improving the Algorithm for Extracting Debris-Covered Glaciers in Optical Satellite Images [J]. Computer Engineering, 2025, 51(2): 269-277.
[13]	ZHOU Jiawen, ZHENG Xiaoying, ZHU Yongxin, LIN Simin, CHEN Lingyao, ZENG Hongbin, GUO Yu, WANG Xinying. Myocardial Ischemia Image Classification via Fusion of Multi-Head Self-Attention and Bilinear Pooling [J]. Computer Engineering, 2025, 51(11): 246-257.
[14]	XU Lei, ZENG Yan, YUAN Junfeng, YUE Lupeng, YIN Yuyu, ZHANG Jilin, XUE Meiting, HAN Meng. Time Series Imputation Based on Self-Attention Mechanism [J]. Computer Engineering, 2025, 51(11): 90-99.
[15]	LIU Zhong, TANG Hong, WANG Ningzhe, ZHU Chuanrun. Text Summarization Method Incorporating RNN and Sparse Self-Attention [J]. Computer Engineering, 2025, 51(1): 312-320.

Please choose a citation manager

Content to export