基于深度学习的CT-MR图像联合配准分割方法

doi:10.19678/j.issn.1000-3428.0065678

计算机工程 ›› 2023, Vol. 49 ›› Issue (9): 234-245. doi: 10.19678/j.issn.1000-3428.0065678

基于深度学习的CT-MR图像联合配准分割方法

洪犇¹^,², 钱旭升², 申明磊¹, 胡冀苏², 耿辰², 戴亚康², 周志勇²^,*

1. 南京理工大学电子工程与光电技术学院, 南京 210094
2. 中国科学院苏州生物医学工程技术研究所, 江苏苏州 215163

收稿日期:2022-09-05 出版日期:2023-09-15 发布日期:2023-09-14
通讯作者: 周志勇
作者简介:
洪犇（1997—），男，硕士，主研方向为图像处理
钱旭升，硕士
申明磊，副研究员、博士
胡冀苏，博士
耿辰，博士
戴亚康, 研究员、博士
基金资助:
中国科学院青年创新促进会项目(2021324); 江苏省重点研发计划项目(BE2022049-2); 江苏省重点研发计划项目(BE2021053); 江苏省重点研发计划项目(BE2020625); 江苏省重点研发计划项目(BE2021612); 江苏省卫生健康委医学科研项目(M2020068); 苏州市医疗卫生科技创新项目(SKY2021031); 苏州市科技计划项目(SS202054); 丽水市科技计划项目(2020ZDYF09)

Joint Registration and Segmentation Method of CT-MR Images Based on Deep Learning

Ben HONG¹^,², Xusheng QIAN², Minglei SHEN¹, Jisu HU², Chen GENG², Yakang DAI², Zhiyong ZHOU²^,*

1. School of Electronic Engineering and Optoelectronic Technology, Nanjing University of Science and Technology, Nanjing 210094, China
2. Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Science, Suzhou 215163, Jiangsu, China

Received:2022-09-05 Online:2023-09-15 Published:2023-09-14
Contact: Zhiyong ZHOU

摘要/Abstract

摘要：

医学图像配准和分割是医学图像分析中的两项重要任务，将其相结合可以有效提升两者的精度，但现有的单模态图像联合配准分割框架难以适用于多模态图像。针对以上问题，提出基于模态一致性监督和多尺度邻域描述符的CT-MR图像联合配准分割框架，包含一个多模态图像配准网络和两个分割网络。联合配准分割框架利用多模态图像配准产生的形变场在两种模态的分割结果之间建立对应的形变关系，并设计模态一致性监督损失，通过两个分割网络互相监督的方式提升多模态分割的精度。在多模态图像配准网络中，构建多尺度模态独立邻域描述符以增强跨模态信息表征能力，并将该描述符作为结构性损失项加入配准网络，更加准确地约束多模态图像的局部结构对应关系。在118例肝脏CT-MR多模态图像数据集上的实验结果表明，在仅提供30%分割标签的情况下，该方法的肝脏配准Dice相似系数(DSC)达到94.66(±0.84)%，目标配准误差达到5.191(±1.342) mm，CT和MR图像的肝脏分割DSC达到94.68(±0.82)%和94.12(±1.06)%，优于对比的配准方法和分割方法。

关键词: 多模态图像, 配准, 分割, 模态一致性监督, 多尺度邻域描述符

Abstract:

Medical image registration and segmentation are important tasks in medical image analysis.The accuracy of the tasks can be improved effectively by their combination.However, the existing joint registration and segmentation framework of single-modal images is difficult to apply to multi-modal images.To address these problems, a Computed Tomography-Magnetic Resonance(CT-MR) image-based joint registration and segmentation framework based on modality-consistent supervision and a multi-scale modality-independent neighborhood descriptor is proposed.It consists of a multimodal image registration network and two segmentation networks.The deformation field generated by the multi-modal registration is used to establish the corresponding deformation relationship between the segmentation network results of the two modalities.Modality consistency supervision loss is constructed, which improves the accuracy of multi-modal segmentation because the two segmentation networks supervise each other.In the multimodal image registration network, a multi-scale modality-independent neighborhood descriptor is constructed to enhance the representation ability of cross-modal information.The descriptor is added to the registration network as a structural loss term to constrain the local structure correspondence of multimodal images more accurately.Experiments were performed on a dataset of 118 CT-MR multimodal liver images.When 30% segmentation labels are provided, the Dice Similarity Coefficient(DSC) of liver registration of this method reaches 94.66(±0.84)%, and the Target Registration Error(TRE) reaches 5.191(±1.342) mm.The DSC of liver segmentation of this method reaches 94.68(±0.82)% and 94.12%(±1.06)% in CT and MR images.These results are superior to those of the comparable registration and segmentation method.

Key words: multi-modal images, registration, segmentation, modality consistency supervision, multi-scale neighborhood descriptor

洪犇, 钱旭升, 申明磊, 胡冀苏, 耿辰, 戴亚康, 周志勇. 基于深度学习的CT-MR图像联合配准分割方法[J]. 计算机工程, 2023, 49(9): 234-245.

Ben HONG, Xusheng QIAN, Minglei SHEN, Jisu HU, Chen GENG, Yakang DAI, Zhiyong ZHOU. Joint Registration and Segmentation Method of CT-MR Images Based on Deep Learning[J]. Computer Engineering, 2023, 49(9): 234-245.

https://www.ecice06.com/CN/Y2023/V49/I9/234

图/表 16

图1 基于配准与分割的联合优化网络框架

Fig.1 Joint optimization network framework based on registration and segmentation

图2 配准与分割的交替训练过程

Fig.2 Alternating training process of registration and segmentation

图3 配准子网络结构

Fig.3 Structure of registration subnetwork

图4 分割子网络结构

Fig.4 Structure of segmentation subnetwork

图5 msMIND三维展示

Fig.5 Presentation of msMIND in 3D

图6 msMIND二维展示

Fig.6 Presentation of msMIND in 2D

图7 CT-MR图像展示

Fig.7 Presentation of CT-MR images

图8 不同方法的配准结果展示

Fig.8 Presentation of registration results with different methods

图9 不同方法分割结果展示

Fig.9 Presentation of segmentation results with different methods

图10 不同标签占比下分割与配准的DSC曲线

Fig.10 DSC curve of segmentation registration with different proportion labels

参考文献 39

1	DING W B, LI L, ZHUANG X H, et al. Cross-modality multi-atlas segmentation using deep neural networks. Berlin, Germany: Springer, 2020.
2	CAO X H, YANG J H, WANG L, et al. Deep learning based inter-modality image registration supervised by intra-modality similarity[C]//Proceedings of International Conference on Machine Learning in Medical Imaging. Berlin, Germany: Springer, 2018: 55-63.
3	陈辰, 周拥军, 李元祥, 等. 基于U-Net分割与HEIV模型的遥感图像配准方法. 计算机工程, 2019, 45 (11): 249- 255. doi: 10.19678/j.issn.1000-3428.0052166
	CHEN C, ZHOU Y J, LI Y X, et al. Remote sensing image registration method based on U-Net segmentation and HEIV model. Computer Engineering, 2019, 45 (11): 249- 255. doi: 10.19678/j.issn.1000-3428.0052166
4	王帅坤, 周志勇, 胡冀苏, 等. 基于深度学习的肝脏CT-MR图像无监督配准. 计算机工程, 2023, 49 (1): 223- 233. doi: 10.19678/j.issn.1000-3428.0063999
	WANG S K, ZHOU Z Y, HU J S, et al. Unsupervised registration of liver CT-MR images based on deep learning. Computer Engineering, 2023, 49 (1): 223- 233. doi: 10.19678/j.issn.1000-3428.0063999
5	CHEN X A, XIA Y, RAVIKUMAR N, et al. A deep discontinuity-preserving image registration network[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2021: 46-55.
6	MAHAPATRA D, ANTONY B, SEDAI S M, et al. Deformable medical image registration using generative adversarial networks[C]//Proceedings of the 15th IEEE International Symposium on Biomedical Imaging. Washington D. C., USA: IEEE Press, 2018: 1449-1453.
7	FAN J F, CAO X H, WANG Q, et al. Adversarial learning for mono-or multi-modal registration. Medical Image Analysis, 2019, 58, 101545. doi: 10.1016/j.media.2019.101545
8	XU Z, LUO J E, YAN J P, et al. Adversarial uni- and multi-modal stream networks for multimodal image registration[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2020: 222-232.
9	HEINRICH M P, JENKINSON M, BHUSHAN M, et al. MIND: modality independent neighbourhood descriptor for multi-modal deformable registration. Medical Image Analysis, 2012, 16 (7): 1423- 1435. doi: 10.1016/j.media.2012.05.008
10	BLENDOWSKI M, HANSEN L, HEINRICH M P. Weakly-supervised learning of multi-modal features for regularised iterative descent in 3D image registration. Medical Image Analysis, 2021, 67, 101822. doi: 10.1016/j.media.2020.101822
11	FERRANTE E, DOKANIA P K, SILVA R M, et al. Weakly supervised learning of metric aggregations for deformable image registration. IEEE Journal of Biomedical and Health Informatics, 2019, 23 (4): 1374- 1384. doi: 10.1109/JBHI.2018.2869700
12	HU Y P, MODAT M, GIBSON E, et al. Weakly-supervised convolutional neural networks for multimodal image registration. Medical Image Analysis, 2018, 49, 1- 13. doi: 10.1016/j.media.2018.07.002
13	HU Y P, MODAT M, GIBSON E, et al. Label-driven weakly-supervised learning for multimodal deformable image registration[C]//Proceedings of the 15th IEEE International Symposium on Biomedical Imaging. Washington D. C., USA: IEEE Press, 2018: 1070-1074.
14	DALCA A V, YU E, GOLLAND P, et al. Unsupervised deep learning for Bayesian brain MRI segmentation. Berlin, Germany: Springer, 2019: 356- 365.
15	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2015: 234-241.
16	ZHOU Z W, RAHMAN M M, TAJBAKHSH N, et al. U-Net++: a nested U-Net architecture for medical image segmentation. Berlin, Germany: Springer, 2018: 3- 11.
17	林志洁, 郑秋岚, 梁涌, 等. 基于内卷U-Net的医学图像分割模型. 计算机工程, 2022, 48 (8): 180- 186. URL
	LIN Z J, ZHENG Q L, LIANG Y, et al. Medical image segmentation model based on involution U-Net. Computer Engineering, 2022, 48 (8): 180- 186. URL
18	LI C Z, DONG L, DOU Q, et al. Self-ensembling co-training framework for semi-supervised COVID-19 CT segmentation. IEEE Journal of Biomedical and Health Informatics, 2021, 25 (11): 4140- 4151. doi: 10.1109/JBHI.2021.3103646
19	LI B, NIESSEN W J, KLEIN S, et al. A hybrid deep learning framework for integrated segmentation and registration: evaluation on longitudinal white matter tract changes. Berlin, Germany: Springer, 2019: 645- 653.
20	BITARAFAN A, NIKDAN M, BAGHSHAH M S. 3D image segmentation with sparse annotation by self-training and internal registration. IEEE Journal of Biomedical and Health Informatics, 2021, 25 (7): 2665- 2672. doi: 10.1109/JBHI.2020.3038847
21	SHAO W, BHATTACHARYA I, SOERENSEN S J C, et al. Weakly supervised registration of prostate MRI and histopathology images[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2021: 98-107.
22	ZHOU B, AUGENFELD Z, CHAPIRO J, et al. Anatomy-guided multimodal registration by learning segmentation without ground truth: application to intraprocedural CBCT/MR liver segmentation and registration. Medical Image Analysis, 2021, 71, 102041. doi: 10.1016/j.media.2021.102041
23	LI B, NIESSEN W J, KLEIN S, et al. Longitudinal diffusion MRI analysis using Segis-Net: a single-step deep-learning framework for simultaneous segmentation and registration. NeuroImage, 2021, 235, 118004. doi: 10.1016/j.neuroimage.2021.118004
24	吕凯, 吴军. 基于B样条和水平集方法的医学图像联合分割与配准. 激光与光电子学进展, 2020, 57 (10): 101007. URL
	LÜ K, WU J. Joint segmentation and registration of medical image based on B-spline and level set method. Laser & Optoelectronics Progress, 2020, 57 (10): 101007. URL
25	ESTIENNE T, VAKALOPOULOU M, CHRISTODOULIDIS S, et al. U-ReSNet: ultimate coupling of registration and segmentation with deep nets[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2019: 310-319.
26	ELMAHDY M S, BELJAARDS L, YOUSEFI S, et al. Joint registration and segmentation via multi-task learning for adaptive radiotherapy of prostate cancer. IEEE Access, 2021, 9, 95551- 95568. doi: 10.1109/ACCESS.2021.3091011
27	XU Z L, NIETHAMMER M. DeepAtlas: joint semi-supervised learning of image registration and segmentation[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2019: 420-429.
28	QIU L, REN H L. U-RSNet: an unsupervised probabilistic model for joint registration and segmentation. Neurocomputing, 2021, 450, 264- 274. doi: 10.1016/j.neucom.2021.04.042
29	QIU L, REN H L. RSegNet: a joint learning framework for deformable registration and segmentation. IEEE Transactions on Automation Science and Engineering, 2022, 19 (3): 2499- 2513. doi: 10.1109/TASE.2021.3087868
30	SU Y X, LAN T, LIU Y H, et al. Language models can see: plugging visual controls in text generation[EB/OL]. [2022-08-10]. https://arxiv.org/abs/2205.02655.
31	ROY A G, NAVAB N, WACHINGER C. Concurrent spatial and channel 'squeeze & excitation' in fully convolutional networks[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2018: 421-429.
32	ISENSEE F, JAEGER P F, KOHL S A A, et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 2021, 18 (2): 203- 211.
33	KIM S, MIN D B, LIN S, et al. Dense cross-modal correspondence estimation with the deep self-correlation descriptor. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (7): 2345- 2359. doi: 10.1109/TPAMI.2020.2965528
34	SHECHTMAN E, IRANI M. Matching local self-similarities across images and videos[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2007: 1-8.
35	HEINRICH M P, JENKINSON M, PAPIEŻ B W, et al. Towards realtime multimodal fusion for image-guided interventions using self-similarities[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2013: 187-194.
36	YANG H R, SUN J, CARASS A, et al. Unsupervised MR-to-CT synthesis using structure-constrained CycleGAN. IEEE Transactions on Medical Imaging, 2020, 39 (12): 4249- 4261.
37	KLEIN S, STARING M, MURPHY K, et al. Elastix: a toolbox for intensity-based medical image registration. IEEE Transactions on Medical Imaging, 2010, 29 (1): 196- 205.
38	BALAKRISHNAN G, ZHAO A, SABUNCU M R, et al. VoxelMorph: a learning framework for deformable medical image registration. IEEE Transactions on Medical Imaging, 2019, 38 (8): 1788- 1800.
39	CICEK Ö, ABDULKADIR A, LIENKAMP S S, et al. 3D U-Net: learning dense volumetric segmentation from sparse annotation[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2016: 424-432.

[1]	林畅, 郭伟, 任哲聪, 金海波. 基于Transformer的目标跟踪与分割统一算法[J]. 计算机工程, 2024, 50(9): 130-141.
[2]	郭敏, 张熙涵, 李阳. 融合注意力的教师互一致性半监督医学图像分割[J]. 计算机工程, 2024, 50(9): 313-323.
[3]	潘恩元, 钟原, 李平. 联邦异质性数据下半监督颈椎MRI分割模型[J]. 计算机工程, 2024, 50(9): 367-376.
[4]	李仲, 冒睿瑞, 王晓龙, 王根一, 安国成. 基于改进PIDNet的水位线检测算法[J]. 计算机工程, 2024, 50(8): 102-112.
[5]	曾湄, 王逸涵, 雷志伟, 刘雪垠, 李柏林. 基于自监督学习的葡萄实例去重叠遮挡算法[J]. 计算机工程, 2024, 50(8): 216-228.
[6]	赵婉秋, 张俊虎, 李海涛. 用于建筑物分割的平行结构特征融合网络[J]. 计算机工程, 2024, 50(8): 239-248.
[7]	闵莉, 董冰洁, 安冬. 基于多注意力机制与跨特征融合的语义分割算法[J]. 计算机工程, 2024, 50(8): 282-289.
[8]	高爽, 史轶伦, 徐巧枝, 于磊. 基于对比学习的非对称编解码结构的心脏MRI分割研究[J]. 计算机工程, 2024, 50(8): 290-300.
[9]	逯焕宇, 张永宏, 马光义, 谢东林, 田伟. 基于半监督对抗学习的遥感图像水体提取[J]. 计算机工程, 2024, 50(7): 251-263.
[10]	肖慈, 徐杨, 张永丹, 冯明文, 黄易仟. 结合注意力和低光增强的夜间语义分割[J]. 计算机工程, 2024, 50(7): 271-281.
[11]	陈晓玉, 沈晨, 沈阅, 孔德明. 基于改进SwiftNet的堆场图像实时分割网络[J]. 计算机工程, 2024, 50(6): 296-303.
[12]	周秦源, 邓越平, 张磊, 张陈, 卢日荣, 胡贤哲. 融合光流与多视角几何的动态视觉SLAM系统[J]. 计算机工程, 2024, 50(5): 250-259.
[13]	杨玉聃, 张俊华, 刘云凤. 基于三维循环残差卷积的脊柱CT图像分割[J]. 计算机工程, 2024, 50(4): 237-246.
[14]	胡帅, 李华玲, 郝德琛. 改进U-Net的多级边缘增强医学图像分割网络[J]. 计算机工程, 2024, 50(4): 286-293.
[15]	王安政, 党建武, 岳彪, 杨景玉. 基于位置信息和注意力机制的路面裂缝检测[J]. 计算机工程, 2024, 50(4): 303-312.

选择文件类型/文献管理软件名称

选择包含的内容

基于深度学习的CT-MR图像联合配准分割方法

Joint Registration and Segmentation Method of CT-MR Images Based on Deep Learning

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 16

参考文献 39

相关文章 15

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于深度学习的CT-MR图像联合配准分割方法

Joint Registration and Segmentation Method of CT-MR Images Based on Deep Learning

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 16

参考文献 39

相关文章 15

编辑推荐

Metrics

本文评价