基于深度学习的CT-MR图像联合配准分割方法

doi:10.19678/j.issn.1000-3428.0065678

计算机工程 ›› 2023, Vol. 49 ›› Issue (9): 234-245. doi: 10.19678/j.issn.1000-3428.0065678

基于深度学习的CT-MR图像联合配准分割方法

洪犇¹^,², 钱旭升², 申明磊¹, 胡冀苏², 耿辰², 戴亚康², 周志勇²^,*

1. 南京理工大学电子工程与光电技术学院, 南京 210094
2. 中国科学院苏州生物医学工程技术研究所, 江苏苏州 215163

收稿日期:2022-09-05 出版日期:2023-09-15 发布日期:2023-09-14
通讯作者: 周志勇
作者简介:
洪犇（1997—），男，硕士，主研方向为图像处理
钱旭升，硕士
申明磊，副研究员、博士
胡冀苏，博士
耿辰，博士
戴亚康, 研究员、博士
基金资助:
中国科学院青年创新促进会项目(2021324); 江苏省重点研发计划项目(BE2022049-2); 江苏省重点研发计划项目(BE2021053); 江苏省重点研发计划项目(BE2020625); 江苏省重点研发计划项目(BE2021612); 江苏省卫生健康委医学科研项目(M2020068); 苏州市医疗卫生科技创新项目(SKY2021031); 苏州市科技计划项目(SS202054); 丽水市科技计划项目(2020ZDYF09)

Joint Registration and Segmentation Method of CT-MR Images Based on Deep Learning

Ben HONG¹^,², Xusheng QIAN², Minglei SHEN¹, Jisu HU², Chen GENG², Yakang DAI², Zhiyong ZHOU²^,*

1. School of Electronic Engineering and Optoelectronic Technology, Nanjing University of Science and Technology, Nanjing 210094, China
2. Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Science, Suzhou 215163, Jiangsu, China

Received:2022-09-05 Online:2023-09-15 Published:2023-09-14
Contact: Zhiyong ZHOU

摘要/Abstract

摘要：

医学图像配准和分割是医学图像分析中的两项重要任务，将其相结合可以有效提升两者的精度，但现有的单模态图像联合配准分割框架难以适用于多模态图像。针对以上问题，提出基于模态一致性监督和多尺度邻域描述符的CT-MR图像联合配准分割框架，包含一个多模态图像配准网络和两个分割网络。联合配准分割框架利用多模态图像配准产生的形变场在两种模态的分割结果之间建立对应的形变关系，并设计模态一致性监督损失，通过两个分割网络互相监督的方式提升多模态分割的精度。在多模态图像配准网络中，构建多尺度模态独立邻域描述符以增强跨模态信息表征能力，并将该描述符作为结构性损失项加入配准网络，更加准确地约束多模态图像的局部结构对应关系。在118例肝脏CT-MR多模态图像数据集上的实验结果表明，在仅提供30%分割标签的情况下，该方法的肝脏配准Dice相似系数(DSC)达到94.66(±0.84)%，目标配准误差达到5.191(±1.342) mm，CT和MR图像的肝脏分割DSC达到94.68(±0.82)%和94.12(±1.06)%，优于对比的配准方法和分割方法。

关键词: 多模态图像, 配准, 分割, 模态一致性监督, 多尺度邻域描述符

Abstract:

Medical image registration and segmentation are important tasks in medical image analysis.The accuracy of the tasks can be improved effectively by their combination.However, the existing joint registration and segmentation framework of single-modal images is difficult to apply to multi-modal images.To address these problems, a Computed Tomography-Magnetic Resonance(CT-MR) image-based joint registration and segmentation framework based on modality-consistent supervision and a multi-scale modality-independent neighborhood descriptor is proposed.It consists of a multimodal image registration network and two segmentation networks.The deformation field generated by the multi-modal registration is used to establish the corresponding deformation relationship between the segmentation network results of the two modalities.Modality consistency supervision loss is constructed, which improves the accuracy of multi-modal segmentation because the two segmentation networks supervise each other.In the multimodal image registration network, a multi-scale modality-independent neighborhood descriptor is constructed to enhance the representation ability of cross-modal information.The descriptor is added to the registration network as a structural loss term to constrain the local structure correspondence of multimodal images more accurately.Experiments were performed on a dataset of 118 CT-MR multimodal liver images.When 30% segmentation labels are provided, the Dice Similarity Coefficient(DSC) of liver registration of this method reaches 94.66(±0.84)%, and the Target Registration Error(TRE) reaches 5.191(±1.342) mm.The DSC of liver segmentation of this method reaches 94.68(±0.82)% and 94.12%(±1.06)% in CT and MR images.These results are superior to those of the comparable registration and segmentation method.

Key words: multi-modal images, registration, segmentation, modality consistency supervision, multi-scale neighborhood descriptor

洪犇, 钱旭升, 申明磊, 胡冀苏, 耿辰, 戴亚康, 周志勇. 基于深度学习的CT-MR图像联合配准分割方法[J]. 计算机工程, 2023, 49(9): 234-245.

Ben HONG, Xusheng QIAN, Minglei SHEN, Jisu HU, Chen GENG, Yakang DAI, Zhiyong ZHOU. Joint Registration and Segmentation Method of CT-MR Images Based on Deep Learning[J]. Computer Engineering, 2023, 49(9): 234-245.

http://www.ecice06.com/CN/Y2023/V49/I9/234

图/表 16

图1 基于配准与分割的联合优化网络框架

Fig.1 Joint optimization network framework based on registration and segmentation

图2 配准与分割的交替训练过程

Fig.2 Alternating training process of registration and segmentation

图3 配准子网络结构

Fig.3 Structure of registration subnetwork

图4 分割子网络结构

Fig.4 Structure of segmentation subnetwork

图5 msMIND三维展示

Fig.5 Presentation of msMIND in 3D

图6 msMIND二维展示

Fig.6 Presentation of msMIND in 2D

图7 CT-MR图像展示

Fig.7 Presentation of CT-MR images

图8 不同方法的配准结果展示

Fig.8 Presentation of registration results with different methods

图9 不同方法分割结果展示

Fig.9 Presentation of segmentation results with different methods

图10 不同标签占比下分割与配准的DSC曲线

Fig.10 DSC curve of segmentation registration with different proportion labels

参考文献 39

1	DING W B, LI L, ZHUANG X H, et al. Cross-modality multi-atlas segmentation using deep neural networks. Berlin, Germany: Springer, 2020.
2	CAO X H, YANG J H, WANG L, et al. Deep learning based inter-modality image registration supervised by intra-modality similarity[C]//Proceedings of International Conference on Machine Learning in Medical Imaging. Berlin, Germany: Springer, 2018: 55-63.
3	陈辰, 周拥军, 李元祥, 等. 基于U-Net分割与HEIV模型的遥感图像配准方法. 计算机工程, 2019, 45 (11): 249- 255. doi: 10.19678/j.issn.1000-3428.0052166
	CHEN C, ZHOU Y J, LI Y X, et al. Remote sensing image registration method based on U-Net segmentation and HEIV model. Computer Engineering, 2019, 45 (11): 249- 255. doi: 10.19678/j.issn.1000-3428.0052166
4	王帅坤, 周志勇, 胡冀苏, 等. 基于深度学习的肝脏CT-MR图像无监督配准. 计算机工程, 2023, 49 (1): 223- 233. doi: 10.19678/j.issn.1000-3428.0063999
	WANG S K, ZHOU Z Y, HU J S, et al. Unsupervised registration of liver CT-MR images based on deep learning. Computer Engineering, 2023, 49 (1): 223- 233. doi: 10.19678/j.issn.1000-3428.0063999
5	CHEN X A, XIA Y, RAVIKUMAR N, et al. A deep discontinuity-preserving image registration network[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2021: 46-55.
6	MAHAPATRA D, ANTONY B, SEDAI S M, et al. Deformable medical image registration using generative adversarial networks[C]//Proceedings of the 15th IEEE International Symposium on Biomedical Imaging. Washington D. C., USA: IEEE Press, 2018: 1449-1453.
7	FAN J F, CAO X H, WANG Q, et al. Adversarial learning for mono-or multi-modal registration. Medical Image Analysis, 2019, 58, 101545. doi: 10.1016/j.media.2019.101545
8	XU Z, LUO J E, YAN J P, et al. Adversarial uni- and multi-modal stream networks for multimodal image registration[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2020: 222-232.
9	HEINRICH M P, JENKINSON M, BHUSHAN M, et al. MIND: modality independent neighbourhood descriptor for multi-modal deformable registration. Medical Image Analysis, 2012, 16 (7): 1423- 1435. doi: 10.1016/j.media.2012.05.008
10	BLENDOWSKI M, HANSEN L, HEINRICH M P. Weakly-supervised learning of multi-modal features for regularised iterative descent in 3D image registration. Medical Image Analysis, 2021, 67, 101822. doi: 10.1016/j.media.2020.101822
11	FERRANTE E, DOKANIA P K, SILVA R M, et al. Weakly supervised learning of metric aggregations for deformable image registration. IEEE Journal of Biomedical and Health Informatics, 2019, 23 (4): 1374- 1384. doi: 10.1109/JBHI.2018.2869700
12	HU Y P, MODAT M, GIBSON E, et al. Weakly-supervised convolutional neural networks for multimodal image registration. Medical Image Analysis, 2018, 49, 1- 13. doi: 10.1016/j.media.2018.07.002
13	HU Y P, MODAT M, GIBSON E, et al. Label-driven weakly-supervised learning for multimodal deformable image registration[C]//Proceedings of the 15th IEEE International Symposium on Biomedical Imaging. Washington D. C., USA: IEEE Press, 2018: 1070-1074.
14	DALCA A V, YU E, GOLLAND P, et al. Unsupervised deep learning for Bayesian brain MRI segmentation. Berlin, Germany: Springer, 2019: 356- 365.
15	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2015: 234-241.
16	ZHOU Z W, RAHMAN M M, TAJBAKHSH N, et al. U-Net++: a nested U-Net architecture for medical image segmentation. Berlin, Germany: Springer, 2018: 3- 11.
17	林志洁, 郑秋岚, 梁涌, 等. 基于内卷U-Net的医学图像分割模型. 计算机工程, 2022, 48 (8): 180- 186. URL
	LIN Z J, ZHENG Q L, LIANG Y, et al. Medical image segmentation model based on involution U-Net. Computer Engineering, 2022, 48 (8): 180- 186. URL
18	LI C Z, DONG L, DOU Q, et al. Self-ensembling co-training framework for semi-supervised COVID-19 CT segmentation. IEEE Journal of Biomedical and Health Informatics, 2021, 25 (11): 4140- 4151. doi: 10.1109/JBHI.2021.3103646
19	LI B, NIESSEN W J, KLEIN S, et al. A hybrid deep learning framework for integrated segmentation and registration: evaluation on longitudinal white matter tract changes. Berlin, Germany: Springer, 2019: 645- 653.
20	BITARAFAN A, NIKDAN M, BAGHSHAH M S. 3D image segmentation with sparse annotation by self-training and internal registration. IEEE Journal of Biomedical and Health Informatics, 2021, 25 (7): 2665- 2672. doi: 10.1109/JBHI.2020.3038847
21	SHAO W, BHATTACHARYA I, SOERENSEN S J C, et al. Weakly supervised registration of prostate MRI and histopathology images[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2021: 98-107.
22	ZHOU B, AUGENFELD Z, CHAPIRO J, et al. Anatomy-guided multimodal registration by learning segmentation without ground truth: application to intraprocedural CBCT/MR liver segmentation and registration. Medical Image Analysis, 2021, 71, 102041. doi: 10.1016/j.media.2021.102041
23	LI B, NIESSEN W J, KLEIN S, et al. Longitudinal diffusion MRI analysis using Segis-Net: a single-step deep-learning framework for simultaneous segmentation and registration. NeuroImage, 2021, 235, 118004. doi: 10.1016/j.neuroimage.2021.118004
24	吕凯, 吴军. 基于B样条和水平集方法的医学图像联合分割与配准. 激光与光电子学进展, 2020, 57 (10): 101007. URL
	LÜ K, WU J. Joint segmentation and registration of medical image based on B-spline and level set method. Laser & Optoelectronics Progress, 2020, 57 (10): 101007. URL
25	ESTIENNE T, VAKALOPOULOU M, CHRISTODOULIDIS S, et al. U-ReSNet: ultimate coupling of registration and segmentation with deep nets[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2019: 310-319.
26	ELMAHDY M S, BELJAARDS L, YOUSEFI S, et al. Joint registration and segmentation via multi-task learning for adaptive radiotherapy of prostate cancer. IEEE Access, 2021, 9, 95551- 95568. doi: 10.1109/ACCESS.2021.3091011
27	XU Z L, NIETHAMMER M. DeepAtlas: joint semi-supervised learning of image registration and segmentation[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2019: 420-429.
28	QIU L, REN H L. U-RSNet: an unsupervised probabilistic model for joint registration and segmentation. Neurocomputing, 2021, 450, 264- 274. doi: 10.1016/j.neucom.2021.04.042
29	QIU L, REN H L. RSegNet: a joint learning framework for deformable registration and segmentation. IEEE Transactions on Automation Science and Engineering, 2022, 19 (3): 2499- 2513. doi: 10.1109/TASE.2021.3087868
30	SU Y X, LAN T, LIU Y H, et al. Language models can see: plugging visual controls in text generation[EB/OL]. [2022-08-10]. https://arxiv.org/abs/2205.02655.
31	ROY A G, NAVAB N, WACHINGER C. Concurrent spatial and channel 'squeeze & excitation' in fully convolutional networks[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2018: 421-429.
32	ISENSEE F, JAEGER P F, KOHL S A A, et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 2021, 18 (2): 203- 211.
33	KIM S, MIN D B, LIN S, et al. Dense cross-modal correspondence estimation with the deep self-correlation descriptor. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (7): 2345- 2359. doi: 10.1109/TPAMI.2020.2965528
34	SHECHTMAN E, IRANI M. Matching local self-similarities across images and videos[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2007: 1-8.
35	HEINRICH M P, JENKINSON M, PAPIEŻ B W, et al. Towards realtime multimodal fusion for image-guided interventions using self-similarities[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2013: 187-194.
36	YANG H R, SUN J, CARASS A, et al. Unsupervised MR-to-CT synthesis using structure-constrained CycleGAN. IEEE Transactions on Medical Imaging, 2020, 39 (12): 4249- 4261.
37	KLEIN S, STARING M, MURPHY K, et al. Elastix: a toolbox for intensity-based medical image registration. IEEE Transactions on Medical Imaging, 2010, 29 (1): 196- 205.
38	BALAKRISHNAN G, ZHAO A, SABUNCU M R, et al. VoxelMorph: a learning framework for deformable medical image registration. IEEE Transactions on Medical Imaging, 2019, 38 (8): 1788- 1800.
39	CICEK Ö, ABDULKADIR A, LIENKAMP S S, et al. 3D U-Net: learning dense volumetric segmentation from sparse annotation[C]//Proceedings of International Conference on Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2016: 424-432.

[1]	苏晓东, 李世洲, 赵佳圆, 亮洪宇, 张玉荣, 徐红岩. 基于多级叠加和注意力机制的图像语义分割[J]. 计算机工程, 2023, 49(9): 265-271,278.
[2]	徐春波, 闫娟, 杨慧斌, 王博, 吴晗. 基于目标检测和语义分割的视觉SLAM算法[J]. 计算机工程, 2023, 49(8): 199-206, 214.
[3]	张天骐, 闻斌, 熊天, 吴超. 基于张量分解与场景分割的鲁棒视频水印算法[J]. 计算机工程, 2023, 49(8): 250-256, 264.
[4]	付雪, 朱良宽, 黄建平, 王璟瑀, ARYSTANRyspayev. 基于改进北方苍鹰优化算法的多阈值图像分割[J]. 计算机工程, 2023, 49(7): 232-241.
[5]	周逸云, 万新军, 胡伏原, 陈昊. 基于联合注意与特征关联的实例分割算法[J]. 计算机工程, 2023, 49(6): 217-226.
[6]	陈倩倩, 秦宝东. 基于SM9的两方协同盲签名方案[J]. 计算机工程, 2023, 49(6): 144-153,161.
[7]	孙超, 朱勇杰, 余林波, 苗隆鑫, 曹勉, 叶力, 郭乃宇. 基于点面特征融合的RGB-D SLAM算法[J]. 计算机工程, 2023, 49(6): 201-207.
[8]	区展华, 李翠然, 杨茜. 基于ANN的能量采集无线传感器网络中继选择策略[J]. 计算机工程, 2023, 49(5): 215-222,230.
[9]	吕炜琨, 魏琳慧, 郑殿, 刘雨, 望育梅. 火星表面影像分割数据集及性能评估[J]. 计算机工程, 2023, 49(5): 262-268.
[10]	孙欣悦, 李庆忠. 基于小波变换的水下显著性目标快速检测算法[J]. 计算机工程, 2023, 49(4): 249-255.
[11]	白俊卿, 韩柏迅, 张丰侠. 基于深度学习的无人机图像语义分割算法研究[J]. 计算机工程, 2023, 49(4): 233-239.
[12]	马素刚, 陈期梅, 侯志强, 杨小宝, 张子贤. 基于密集连接与特征增强的语义分割算法[J]. 计算机工程, 2023, 49(3): 263-270.
[13]	苏鸣方, 胡立坤, 黄润辉. 基于上下文注意力的室外点云语义分割方法[J]. 计算机工程, 2023, 49(3): 248-256.
[14]	王国仲, 彭醇陵, 周徐. 干扰影响下无线携能通信双向中继系统中断性能分析[J]. 计算机工程, 2023, 49(3): 185-191,202.
[15]	范润泽, 刘宇红, 张荣芬, 李景玉. 基于多尺度注意力机制的道路场景语义分割模型[J]. 计算机工程, 2023, 49(2): 288-295.

选择文件类型/文献管理软件名称

选择包含的内容

基于深度学习的CT-MR图像联合配准分割方法

Joint Registration and Segmentation Method of CT-MR Images Based on Deep Learning

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 16

参考文献 39

相关文章 15

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于深度学习的CT-MR图像联合配准分割方法

Joint Registration and Segmentation Method of CT-MR Images Based on Deep Learning

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 16

参考文献 39

相关文章 15

编辑推荐

Metrics

本文评价