Colon Polyp Segmentation Network Based on Multi-task Joint Attention

doi:10.19678/j.issn.1000-3428.0067899

Abstract

Abstract:

Colon polyps have the characteristics of unclear boundaries and varying sizes, colors, and shapes, making it difficult to improve their segmentation performance using deep learning methods. A colon polyp segmentation network CPMJA-Net based on multi-task joint attention is proposed to improve the accuracy of polyp segmentation. To address the problem of Transformers lacking mechanisms to improve local information exchange, a cascaded fusion module is designed to enhance the local feature representation of the network, which aids in the recognition and restoration of polyp edges.Inspired by the multi-head Self-Attention mechanism, a Multi-task Attention Module(MAM) is constructed, and the feature maps obtained from different modules are gradually fused using a progressive fusion method to highlight key information and suppress interference information. A Joint Attention Module(JAM) is designed to use the contour information of advanced features to filter out detailed features that are conducive to edge segmentation from low-level features and aggregate them with the contour of polyps to obtain more accurate edge segmentation results to better aggregate the advanced and low-level features of images. The experimental results show that CPMJA-Net has the best performance of all four public datasets. Compared with the suboptimal algorithm, the mDice coefficient of CPMJA-Net has improved by 0.7, 0.8, 0.4, and 0.4 percentage points on the Kvasir, CVC-CilinicDB, CVC-ColonDB, and ETIS datasets, respectively. In addition, the mean Intersection over Union(mIoU) increased by 1.6, 1.2, 0.6, and 0.5 percentage points, respectively. Experiments have shown that CPMJA-Net improves over-segmentation, compensates for attention mechanism shortcomings, and improves the decoder's ability to recover details.

Key words: detection of intestinal polyps, PVT network, self-attention mechanism, multi-task attention, joint attention

摘要：

结肠息肉具有边界不清晰且大小、颜色、形状各异的特点，使得采用深度学习方法提高其分割性能仍是一项极具挑战性的工作。为提高息肉分割的准确率，提出一种基于多任务联合注意力的结肠息肉分割网络CPMJA-Net。为改善Transformer缺乏机制来增强局部区域信息交换的问题，设计级联融合模块以增强网络的局部特征表示，有助于息肉边缘的识别和恢复。受多头Self-Attention机制的启发，构建多任务注意力模块，采用渐进式融合的方式将不同模块得到的特征图逐步进行融合，以凸显关键信息并抑制干扰信息。为更好聚合图像的高级和低级特征，设计联合注意力模块，利用高级特征的轮廓信息在低级特征中筛选出有利于边缘分割的细节特征，并将其与息肉轮廓聚合起来，得到更加精确的边缘分割结果。实验结果表明，CPMJA-Net在4个公开数据集上的性能表现均为最优，与次优的算法相比，CPMJA-Net的mDice系数分别在Kvasir、CVC-CilinicDB、CVC-ColonDB和ETIS数据集上提升0.7、0.8、0.4、0.4个百分点，平均交并比（mIoU）也分别提升1.6、1.2、0.6、0.5个百分点，其能改善过分割问题，弥补注意力机制的不足，提升解码器的细节恢复能力。

关键词: 肠道息肉检测, PVT网络, 自注意力机制, 多任务注意力, 联合注意力

Xiangzhen GUO, Sitong LI, Rui LU, Sen GUO, Xuerong CUI, Gang YANG. Colon Polyp Segmentation Network Based on Multi-task Joint Attention[J]. Computer Engineering, 2024, 50(2): 327-336.

郭祥振, 李思潼, 卢锐, 郭森, 崔学荣, 杨钢. 基于多任务联合注意力的结肠息肉分割网络[J]. 计算机工程, 2024, 50(2): 327-336.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0067899

http://www.ecice06.com/EN/Y2024/V50/I2/327

Figures/Tables 9

Fig.1 Overall structure of CPMJA-Net network

Fig.2 Structure of cascading fusion module

Fig.3 Structure of multi-task attention module

Fig.4 Structure of foreground attention unit

Fig.5 Structure of joint attention module

Fig.6 Comparison of prediction results images among different algorithms

References 34

1	田传鑫, 赵磊. 结直肠癌及结直肠癌肝转移流行病学特点. 中华肿瘤防治杂志, 2021, 28 (13): 1033- 1038. doi: 10.16073/j.cnki.cjcpt.2021.13.12
	TIAN C X, ZHAO L. Epidemiological characteristics of colorectal cancer and colorectal liver metastasis. Chinese Journal of Cancer Prevention and Treatment, 2021, 28 (13): 1033- 1038. doi: 10.16073/j.cnki.cjcpt.2021.13.12
2	李道娟, 李倩, 贺宇彤. 结直肠癌流行病学趋势. 肿瘤防治研究, 2015, 42 (3): 305- 310. doi: 10.3971/j.issn.1000-8578.2015.03.020
	LI D J, LI Q, HE Y T. Epidemiological trends of colorectal cancer. Cancer Research on Prevention and Treatment, 2015, 42 (3): 305- 310. doi: 10.3971/j.issn.1000-8578.2015.03.020
3	AHN S B, HAN D S, BAE J H, et al. The miss rate for colorectal adenoma determined by quality-adjusted, back-to-back colonoscopies. Gut and Liver, 2012, 6 (1): 64- 70. doi: 10.5009/gnl.2012.6.1.64
4	FIORI M, MUSÉ P, SAPIRO G. A complete system for candidate polyps detection in virtual colonoscopy. International Journal of Pattern Recognition and Artificial Intelligence, 2014, 28 (7): 1460014. doi: 10.1142/S0218001414600143
5	MAMONOV A V, FIGUEIREDO I N, FIGUEIREDO P N, et al. Automated polyp detection in colon capsule endoscopy. IEEE Transactions on Medical Imaging, 2014, 33 (7): 1488- 1502. doi: 10.1109/TMI.2014.2314959
6	BERNAL J, SÁNCHEZ F, FERNÁNDEZ-ESPARRACH G, et al. WM-DOVA maps for accurate polyp highlighting in colonoscopy: validation vs. saliency maps from physicians. Computerized Medical Imaging and Graphics, 2015, 43, 99- 111. doi: 10.1016/j.compmedimag.2015.02.007
7	SASMAL P, IWAHORI Y, BHUYAN M K, et al. Active contour segmentation of polyps in capsule endoscopic images[C]//Proceedings of International Conference on Signals and Systems. Washington D. C., USA: IEEE Press, 2018: 201-204.
8	任莉莉, 边璇, 王光磊, 等. 基于深度学习的息肉分割网络GLIA-Net. 计算机工程, 2022, 48 (12): 248- 254. URL
	REN L L, BIAN X, WANG G L, et al. Polyp segmentation network GLIA-Net based on deep learning. Computer Engineering, 2022, 48 (12): 248- 254. URL
9	郑秋梅, 徐林康, 王风华, 等. 基于改进自注意力机制的金字塔场景解析网络. 计算机工程, 2023, 49 (1): 242- 249. URL
	ZHENG Q M, XU L K, WANG F H, et al. Pyramid scene parsing network based on improved self-attention mechanism. Computer Engineering, 2023, 49 (1): 242- 249. URL
10	袁单飞, 陈慈发, 董方敏. 基于多尺度分割的图像识别残差网络研究. 计算机工程, 2022, 48 (5): 258-262, 271. URL
	YUAN D F, CHEN C F, DONG F M. Research on residual network of image recognition based on multiscale split. Computer Engineering, 2022, 48 (5): 258-262, 271. URL
11	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 3431-3440.
12	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer: 234-241.
13	BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39 (12): 2481- 2495. doi: 10.1109/TPAMI.2016.2644615
14	ZHOU Z W, SIDDIQUEE M M R, TAJBAKHSH N, et al. UNet++: a nested U-Net architecture for medical image segmentation[C]//Proceedings of International Workshop on Deep Learning in Medical Image Analysis. Berlin, Germany: Springer, 2022: 1-10.
15	JHA D, SMEDSRUD P H, MICHAEL A R, et al. ResUNet++: an advanced architecture for medical image segmentation[C]//Proceedings of International Symposium on Multimedia. Washington D. C., USA: IEEE Press, 2019: 1-10.
16	FAN D P, JI G P, ZHOU T, et al. PraNet: parallel reverse attention network for polyp segmentation[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. 2020: 1-10.
17	吉淑滢, 肖志勇. 融合上下文和多尺度特征的胸部多器官分割. 中国图象图形学报, 2021, 26 (9): 2135- 2145. URL
	JI S Y, XIAO Z Y. Integrated context and multi-scale features in thoracic organs segmentation. Journal of Image and Graphics, 2021, 26 (9): 2135- 2145. URL
18	WEI J, HU Y W, ZHANG R M, et al. Shallow attention network for polyp segmentation[EB/OL]. [2023-05-15]. https://arxiv.org/abs/2108.00882v1.
19	DONG B, WANG W H, FAN D P, et al. Polyp-PVT: polyp segmentation with pyramid vision transformers[EB/OL]. [2023-05-15]. https://arxiv.org/abs/2108.06932v5.
20	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2010: 5998-6008.
21	WANG W H, XIE E Z, LI X A, et al. Pyramid vision transformer: a versatile backbone for dense prediction without convolutions[C]//Proceedings of International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 1-10.
22	杨建秀. 无人机视角下车辆目标检测的算法研究[D]. 西安: 西安电子科技大学, 2021.
	YANG J X. Research on algorithm of vehicle target detection from the perspective of UAV[D]. Xi'an: Xidian University, 2021. (in Chinese)
23	李金星, 孙俊, 李超, 等. 融合多头注意力机制的新冠肺炎联合诊断与分割. 中国图象图形学报, 2022, 27 (12): 3651- 3662. URL
	LI J X, SUN J, LI C, et al. A MHA-based integrated diagnosis and segmentation method for COVID-19 pandemic. Journal of Image and Graphics, 2022, 27 (12): 3651- 3662. URL
24	LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 1-10.
25	SILVA J, HISTACE A, ROMAIN O, et al. Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer. International Journal of Computer Assisted Radiology and Surgery, 2014, 9 (2): 283- 293. doi: 10.1007/s11548-013-0926-3
26	TAJBAKHSH N, GURUDU S R, LIANG J M. Automated polyp detection in colonoscopy videos using shape and context information. IEEE Transactions on Medical Imaging, 2016, 35 (2): 630- 644. doi: 10.1109/TMI.2015.2487997
27	BERNAL J, SÁNCHEZ F J, FERNÁNDEZ-ESPARRACH G, et al. WM-DOVA maps for accurate polyp highlighting in colonoscopy: validation vs. saliency maps from physicians. Computerized Medical Imaging and Graphics, 2015, 43, 99- 111. doi: 10.1016/j.compmedimag.2015.02.007
28	JHA D, SMEDSRUD P H, RIEGLER M H, et al. Kvasir-SEG: a segmented polyp dataset[C]//Proceedings of International Conference on Multimedia Modeling. Berlin, Germany: Springer, 2020: 451-462.
29	董波, 周燕, 王永雄. 基于渐进结构感受野和全局注意力的显著性检测. 电子科技, 2021, 34 (1): 23- 30. doi: 10.16180/j.cnki.issn1007-7820.2021.01.005
	DONG B, ZHOU Y, WANG Y X. Saliency detection by progressive structural receptive field and global attention. Electronic Science and Technology, 2021, 34 (1): 23- 30. doi: 10.16180/j.cnki.issn1007-7820.2021.01.005
30	FAN D P, JI G P, QIN X, et al. Cognitive vision inspired object segmentation metric and loss function. Scientia Sinica Informationis, 2021, 51 (9): 1475. doi: 10.1360/SSI-2020-0370
31	FANG Y Q, CHEN C, YUAN Y X, et al. Selective feature aggregation network with area-boundary constraints for polyp segmentation[C]//Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin, Germany: Springer, 2019: 302-310.
32	YIN Z J, LIANG K M, MA Z Y, et al. Duplex contextual relation network for polyp segmentation[C]//Proceedings of the 19th International Symposium on Biomedical Imaging. Washington D. C., USA: IEEE Press, 2022: 1-5.
33	HUANG C H, WU H Y, LIN Y L. HarDNet-MSEG: a simple encoder-decoder polyp segmentation neural network that achieves over 0.9 mean dice and 86 fps[EB/OL]. [2023-05-15]. https://arxiv.org/abs/2101.07172v2.
34	LOU A, GUAN S Y, LOEW M, et al. CaraNet: context axial reverse attention network for segmentation of small medical objects[EB/OL]. [2023-05-15]. https://arxiv.org/abs/2108.07368v1.

[1]	Haochen XU, Manhua LIU. Facial Landmark Detection Based on Hierarchical Self-Attention Network [J]. Computer Engineering, 2024, 50(2): 239-246.
[2]	Zhengjia WANG, Feifei HU, Chengjuan ZHANG, Zhuo LEI, Tao HE. Adaptive Window Stereo Matching Algorithm with Lightweight Transformer [J]. Computer Engineering, 2024, 50(2): 256-265.
[3]	Hao LIU, Honglan WU, Yuxuan FANG. Efficient Human Pose Estimation Combining Global Contextual Information [J]. Computer Engineering, 2023, 49(7): 102-109.
[4]	ZHOU Yiyun, WAN Xinjun, HU Fuyuan, CHEN Hao. Instance Segmentation Algorithm Based on Joint Attention and Feature Association [J]. Computer Engineering, 2023, 49(6): 217-226.
[5]	SHEN Xueli, HAN Qianwen. Field-Aware Click-Through Rate Prediction Model Based on Attention Mechanism [J]. Computer Engineering, 2023, 49(3): 80-86,94.
[6]	LIU Hang, YIN Xin, CHEN Jie, LUO Heng. Multi-Dimensional Time-Series Prediction Based on Hybrid Network Models [J]. Computer Engineering, 2023, 49(1): 121-129.
[7]	ZHENG Qiumei, XU Linkang, WANG Fenghua, LIN Chao. Pyramid Scene Parsing Network Based on Improved Self-Attention Mechanism [J]. Computer Engineering, 2023, 49(1): 242-249.
[8]	ZHAO Guochuan, WANG Heng, ZHANG Hua, PANG Jie, ZHOU Jian. Hydropower Complex Defect Recognition Method Based on Pure Self-Attention [J]. Computer Engineering, 2022, 48(9): 277-285.
[9]	WU Di, WANG Ziyu, ZHAO Weichao. ELMo-CNN-BiGRU Dual-Channel Text Sentiment Classification Model [J]. Computer Engineering, 2022, 48(8): 105-112.
[10]	CHEN Kejia, LIU Hui. Chinese Text Classification Method Based on Improved BiGRU-CNN [J]. Computer Engineering, 2022, 48(5): 59-66,73.
[11]	LIANG Zhibin, ZHAO Wenyi, LI Lingqiao, YANG Huihua. Attachment Removal Network Based on Micro Image of Tool Edge [J]. Computer Engineering, 2022, 48(4): 247-254,261.
[12]	SHI Zhantang, MA Yupeng, ZHAO Fan, MA Bo. Chinese Named Entity Recognition Based on CNN-Head Transformer Encoder [J]. Computer Engineering, 2022, 48(10): 73-80.
[13]	ZHAO Yanan, LIU Yuan, SONG She. Financial News Polarity Analysis Fusing with Multi-Head Self-Attention Mechanism [J]. Computer Engineering, 2020, 46(8): 85-92.
[14]	DUAN Dagao, LIANG Shaohu, ZHAO Zhendong, HAN Zhongming. Prediction Model of Chinese Punctuation Based on Self-Attention Mechanism [J]. Computer Engineering, 2020, 46(5): 291-297.

Please choose a citation manager

Content to export