Unsupervised Registration for Liver CT-MR Images Based on Deep Learning

doi:10.19678/j.issn.1000-3428.0063999

Computer Engineering ›› 2023, Vol. 49 ›› Issue (1): 223-233. doi: 10.19678/j.issn.1000-3428.0063999

• Graphics and Image Processing • Previous Articles Next Articles

Unsupervised Registration for Liver CT-MR Images Based on Deep Learning

WANG Shuaikun^1,2, ZHOU Zhiyong², HU Jisu^1,2, QIAN Xusheng^1,2, GENG Chen², CHEN Guangqiang³, JI Jiansong⁴, DAI Yakang^2,5

1. Division of Life Sciences and Medicine, School of Biomedical Engineering(Suzhou), University of Science and Technology of China, Suzhou, Jiangsu 215163, China;
2. Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Science, Suzhou, Jiangsu 215163, China;
3. The Second Affiliated Hospital of Suzhou University, Suzhou, Jiangsu 215000, China;
4. The Lishui Central Hospital, Lishui, Zhejiang 323000, China;
5. Jinan Guoke Medical Engineering Technology Development Co., Ltd., Jinan 250000, China

Received:2022-02-22 Revised:2022-03-23 Published:2022-07-19

基于深度学习的肝脏CT-MR图像无监督配准

王帅坤^1,2, 周志勇², 胡冀苏^1,2, 钱旭升^1,2, 耿辰², 陈光强³, 纪建松⁴, 戴亚康^2,5

1. 中国科学技术大学 (苏州)生物医学工程学院生命科学与医学部, 江苏苏州 215163;
2. 中国科学院苏州生物医学工程技术研究所, 江苏苏州 215163;
3. 苏州大学附属第二医院, 江苏苏州 215000;
4. 丽水市中心医院, 浙江丽水 323000;
5. 济南国科医工科技发展有限公司, 济南 250000

作者简介:王帅坤(1995-),男,硕士研究生,主研方向为医学图像处理;周志勇,研究员、博士;胡冀苏,博士;钱旭升,硕士;耿辰、陈光强,博士;纪建松,主任医师、博士;戴亚康,研究员、博士。
基金资助:
国家自然科学基金（81971685）；国家重点研发计划（2018YFA0703101）；中国科学院青年创新促进会会员基金（2021324）；江苏省重点研发计划（BE2021053）；苏州市科技计划（SS202054）。

Abstract

Abstract: Multimodal registration is a key step in medical image analysis, which plays an important role in the assisted diagnosis and the image-guided surgical treatment of liver cancer.Aiming at the problems of large computation, long time consuming, and low registration accuracy of traditional iterative multimodal registration, this paper proposes an unsupervised deep learning-based image registration method based on multi-scale deformation fusion and dual-input spatial attention.Using the multi-scale deformation fusion architecture captures different resolution features of images to achieve liver registration in a coarse-to-fine pattern and avoids local optimization.The dual-input spatial attention module is used to extract the discrepant features between images by integrating spatial and text information at different levels in the codec stage and enhancing feature expression.Additionally, a structural information loss is introduced to globally optimize the registration network, which does not require any prior information and achieves an accurate unsupervised registration.Experimental results on liver Computed Tomography-Magnetic Resonance(CT-MR) datasets show that the proposed algorithm achieved an optimal global Dice Similarity Coefficient(DSC) and Target Registration Error(TRE) values of 0.926 1 ±0.018 6 and 6.39 ±3.03 mm, respectively, which is superior to Affine, Elastix, and VoxelMorph amongst other algorithms.In addition, the average registration time of the proposed algorithm is 0.35 ±0.018 s, which is nearly 380 times faster than the Elastix algorithm.Results show that the proposed algorithm demonstrates higher registration accuracy and faster registration speed by accurately extracting features and estimating the regular deformation field.

Key words: deep learning, unsupervised registration, multimodal registration, deformation fusion, structural information loss, spatial attention

摘要： 多模态配准是医学图像分析中的关键环节，在肝癌辅助诊断、图像引导的手术治疗中具有重要作用。针对传统的迭代式肝脏多模态配准计算量大、耗时长、配准精度低等问题，提出一种基于多尺度形变融合和双输入空间注意力的无监督深度学习配准算法。利用多尺度形变融合框架提取不同分辨率的图像特征，实现肝脏的逐阶配准，在提高配准精度的同时避免网络陷入局部最优。采用双输入空间注意力模块在编解码阶段融合不同水平的空间和文本信息提取图像间的差异特征，增强特征表达。引入基于邻域描述符的结构信息损失项进行网络迭代优化，不需要任何先验信息即可实现精确的无监督配准。在临床肝脏CT-MR数据集上的实验结果表明，与传统的Affine、Elastix、VoxelMorph等算法相比，该算法达到最优的DSC值和TRE值，分别为0.926 1±0.018 6和6.39±3.03 mm，其平均配准时间为0.35±0.018 s，相比Elastix算法提升了近380倍，能准确地提取特征及估计规则的形变场，具有较高的配准精度和较快的配准速度。

关键词: 深度学习, 无监督配准, 多模态配准, 形变融合, 结构信息损失, 空间注意力

CLC Number:

TP391

WANG Shuaikun, ZHOU Zhiyong, HU Jisu, QIAN Xusheng, GENG Chen, CHEN Guangqiang, JI Jiansong, DAI Yakang. Unsupervised Registration for Liver CT-MR Images Based on Deep Learning[J]. Computer Engineering, 2023, 49(1): 223-233.

王帅坤, 周志勇, 胡冀苏, 钱旭升, 耿辰, 陈光强, 纪建松, 戴亚康. 基于深度学习的肝脏CT-MR图像无监督配准[J]. 计算机工程, 2023, 49(1): 223-233.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0063999

http://www.ecice06.com/EN/Y2023/V49/I1/223

Figures/Tables 16

References

[1] SUNG H, FERLAY J, SIEGEL R L, et al.Global cancer statistics 2020:GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries[J].CA:a Cancer Journal for Clinicians, 2021, 71(3):209-249.
[2] MAES F, COLLIGNON A, VANDERMEULEN D, et al.Multimodality image registration by maximization of mutual information[J].IEEE Transactions on Medical Imaging, 1997, 16(2):187-198.
[3] WACHINGER C, NAVAB N.Entropy and Laplacian images:structural representations for multi-modal registration[J].Medical Image Analysis, 2012, 16(1):1-17.
[4] HEINRICH M P, JENKINSON M, BHUSHAN M, et al.MIND:modality independent neighbourhood descriptor for multi-modal deformable registration[J].Medical Image Analysis, 2012, 16(7):1423-1435.
[5] ZHOU S K, GREENSPAN H, DAVATZIKOS C, et al.A review of deep learning in medical imaging:imaging traits, technology trends, case studies with progress highlights, and future promises[J].Proceedings of the IEEE, 2021, 109(5):820-838.
[6] SIMONOVSKY M, GUTIÉRREZ-BECKER B, MATEUS D, et al.A deep metric for multimodal registration[C]//Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention.Berlin, Germany:Springer, 2016:10-18.
[7] HASKINS G, KRUECKER J, KRUGER U, et al.Learning deep similarity metric for 3D MR-TRUS image registration[J].International Journal of Computer Assisted Radiology and Surgery, 2019, 14(3):417-425.
[8] GOODFELLOW I, POUGET ABADIE J, MIRZA M, et al.Generative adversarial networks[EB/OL].[2022-01-10].https://arxiv.org/abs/1406.2661?context=cs.LG.
[9] YAN P K, XU S, RASTINEHAD A R, et al.Adversarial image registration with application for MR and TRUS image fusion[C]//Proceedings of International Conference on Machine Learning in Medical Imaging.Berlin, Germany:Springer, 2018:197-204.
[10] MAHAPATRA D, ANTONY B, SEDAI S M, et al.Deformable medical image registration using generative adversarial networks[C]//Proceedings of the 15th International Symposium on Biomedical Imaging.Washington, USA:IEEE Press, 2018:1449-1453.
[11] FAN J F, CAO X H, WANG Q, et al.Adversarial learning for mono- or multi-modal registration[J].Medical Image Analysis, 2019, 58:101545-101556.
[12] DE VOS B D, BERENDSEN F F, VIERGEVER M A, et al.End-to-end unsupervised deformable image registration with a convolutional neural network[C]//Proceedings of Conference on Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support.Berlin, Germany:Springer, 2017:204-212.
[13] ROHÉ M M, DATAR M, HEIMANN T, et al.SVF-Net:learning deformable image registration using shape matching[C]//Proceedings of Conference on Medical Image Computing and Computer Assisted Intervention.Berlin, Germany:Springer, 2017:266-274.
[14] BALAKRISHNAN G, ZHAO A, SABUNCU M R, et al.VoxelMorph:a learning framework for deformable medical image registration[J].IEEE Transactions on Medical Imaging, 2019, 38(8):1788-1800.
[15] HU Y P, MODAT M, GIBSON E, et al.Weakly-supervised convolutional neural networks for multimodal image registration[J].Medical Image Analysis, 2018, 49:1-13.
[16] ZHOU B, AUGENFELD Z, CHAPIRO J, et al.Anatomy-guided multimodal registration by learning segmentation without ground truth:application to intraprocedural CBCT/MR liver segmentation and registration[J].Medical Image Analysis, 2021, 71:102041-102049.
[17] MOK T C W, CHUNG A C S.Large deformation diffeomorphic image registration with laplacian pyramid networks[C]//Proceedings of Conference on Medical Image Computing and Computer Assisted Intervention.Berlin, Germany:Springer, 2020:211-221.
[18] ZHOU Y, PANG S, CHENG J, et al.Unsupervised deformable medical image registration via pyramidal residual deformation fields estimation[EB/OL].[2022-01-08].https://doi.org/10.48550/arXiv.2004.07624.
[19] RONNEBERGER O, FISCHER P, BROX T.U-Net:Convolutional networks for biomedical image segmentation[C]//Proceedings of International Conference on Medical Image Computing and Computer-assisted Intervention.Berlin, Germany:Springer, 2015:234-241.
[20] ROMERA-PAREDES B, TORR P H S.Recurrent instance segmentation[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2016:312-329.
[21] JETLEY S, LORD N A, LEE N, et al.Learn to pay attention[EB/OL].[2022-01-08].https://doi.org/10.48550/arXiv.1804.02391.
[22] WANG X L, GIRSHICK R, GUPTA A, et al.Non-local neural networks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:7794-7803.
[23] OKTAY O, SCHLEMPER J, FOLGOC L L, et al.Attention U-net:learning where to look for the pancreas[EB/OL].[2022-01-08].https://doi.org/10.48550/arXiv.1804.03999.
[24] ZAGORUYKO S, KOMODAKIS N.Paying more attention to attention:improving the performance of convolutional neural networks via attention transfer[EB/OL].[2022-01-08].https://www.semanticscholar.org/paper/Paying-More-Attention-to-Attention%3A-Improving-the-Zagoruyko-Komodakis/f7b032a4df721d4ed2bab97f6acd33d62477b7a5.
[25] YANG H R, SUN J, CARASS A, et al.Unsupervised MR-to-CT synthesis using structure-constrained CycleGAN[J].IEEE Transactions on Medical Imaging, 2020, 39(12):4249-4261.
[26] WEI D M, AHMAD S, HUO J Y, et al.SLIR:Synthesis, localization, inpainting, and registration for image-guided thermal ablation of liver tumors[J].Medical Image Analysis, 2020, 65:101763-101771.
[27] KLEIN S, STARING M, MURPHY K, et al.Elastix:a toolbox for intensity-based medical image registration[J].IEEE Transactions on Medical Imaging, 2009, 29(1):196-205.
[28] PLUIM J P W, MAINTZ J B A, VIERGEVER M A.Mutual-information-based registration of medical images:a survey.[J].IEEE Transactions on Medical Imaging, 2003, 22(8):986-1004.
[29] WANG Z, BOVIK A C, SHEIKH H R, et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing, 2004, 13(4):600-612.

Please choose a citation manager

Content to export