Registration Method of MRI-TRUS Images Based on Joint Learning and Multi-Level Wavelet Feature Pyramid

doi:10.19678/j.issn.1000-3428.0069093

Abstract

Abstract:

The registration of Magnetic Resonance Imaging (MRI) images and TRansrectal UltraSound (TRUS) images is a combination of preoperative MRI image registration and intraoperative ultrasound image, combining the advantages of the two image modes to quickly locate the lesion area. This process plays an important role in assisting diagnosis, puncture, intraoperative navigation, and other medical surgery problems. Owing to the inherent representational differences between the two image modes, with significant intensity distortion and deformation, finding an exact dense correspondence relationship between them remains a challenge. To address this issue, this study proposes a weakly supervised deformable registration network framework based on joint learning and a Multi-level Wavelet Feature Pyramid (MWFP) to align MRI and TRUS images. Joint learning is a framework composed of a pre-trained semi-supervised segmentation network and a registration network. The segmentation and registration networks continue to alternate training, and the segmentation network provides prostate label constraint global registration for the registration network, which effectively solves the problem of insufficient labels in the registration network. MWFP is a registration network composed of multi-resolution wavelets. The multi-scale images generated by the wavelet pyramid filter the relevant noise and reduce the difference in representation between the two modes to improve the ability of the registration network to learn the multi-scale features. In addition, a Multi-Scale Feature Fusion Attention (MSFFM) module is designed in the registration network to further screen the features and provide local dense correspondence for registration. In addition, the deformable segmentation images and segmentation labels provided by the registration network are mixed with the original artificial labels and images, and the pseudo-labels and their images generated by the segmentation network are used as input for the segmentation network for additional training, which further improves the performance of multimodal image segmentation. Results on 642 publicly available prostate MRI images and a TRUS image biopsy dataset show that the proposed registration method achieves optimal Dice Similarity Coefficient (DSC), 95% Hausdorff Distance (HD₉₅), Mutual Information (MI), and Structural Similarity (SSIM) of 81.05%±1.77%, 12.83±1.49 mm, 18.12%±4.63%, and 27.12%±4.63%, respectively, which are superior to those of traditional registration methods and advanced deep learning registration methods. In addition, the average registration time of the proposed method is 0.18±0.02 s, which is nearly 400 times higher than that of the traditional methods. The experimental results show that the proposed registration method can accurately estimate the deformation field between prostate MRI images and TRUS images in real time and has higher registration accuracy and registration speed.

Key words: joint learning, Multi-level Wavelet Feature Pyramid (MWFP), deformable registration, Multi-Scale Feature Fusion Attention (MSFFA) module, semi-supervised segmentation

摘要：

磁共振图像(MRI)和经直肠超声(TRUS)图像的配准是将术前MRI配准在超声图像上，结合两种模态图像的优势，快速定位病灶区域，在辅助诊断、穿刺、术中导航等医学手术中起重要作用。由于这两种图像模式之间固有的表征差异，具有显著的强度失真和变形，因此在这两种图像模式之间寻找精确的密集对应关系面临较大挑战。为此，提出一种基于联合学习和多级小波特征金字塔(MWFP)的弱监督可变形配准网络框架，对MRI和TRUS图像进行对齐。联合学习是基于预训练的半监督分割网络和配准网络组成的框架，在联合学习框架中分割网络和配准网络继续交替训练，分割网络为配准网络提供前列腺标签约束全局配准，有效解决了配准网络中标签不足的问题。MWFP是采用多分辨小波构成的配准网络，小波金字塔生成的多尺度图像过滤了噪声并减小了两种模式图像之间的表征差异，提高配准网络学习多尺度特征的能力，并在配准网络中设计多尺度特征融合注意力(MSFFA)模块，对特征进行更进一步筛选，为配准提供局部密集对应关系。此外，配准网络提供的形变分割图像和分割标签混合原有的人工标注标签和图像及其分割网络生成的伪标签和其图像放入分割网络继续训练，进一步提高多模态图像分割的性能。在642例公开前列腺MRI和TRUS图像活检数据集上的实验结果表明，所提的配准方法达到最优的Dice相似系数(DSC)值、95% Hausdorff距离(HD95)、互信息(MI)值和结构相似性(SSIM)值，分别是81.05%±1.77%、12.83±1.49 mm、18.12%±4.63%和27.12%±4.63%，优于对比的传统配准方法和先进的深度学习配准方法。此外，所提方法的平均配准时间为0.18±0.02 s，比传统的方法提升了近400倍。所提的配准方法能够准确实时地估计前列腺MRI和TRUS图像之间的形变场，具有更高的配准精度和更快的配准速度。

关键词: 联合学习, 多级小波特征金字塔, 可变形配准, 多尺度特征融合注意力模块, 半监督分割

JIANG Honggui, HU Jisu, QIAN Xusheng, ZHENG Yi, ZHOU Zhiyong, DAI Yakang. Registration Method of MRI-TRUS Images Based on Joint Learning and Multi-Level Wavelet Feature Pyramid[J]. Computer Engineering, 2025, 51(10): 270-283.

蒋宏贵, 胡冀苏, 钱旭升, 郑毅, 周志勇, 戴亚康. 基于联合学习和多级小波特征金字塔的MRI-TRUS图像配准方法[J]. 计算机工程, 2025, 51(10): 270-283.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0069093

https://www.ecice06.com/EN/Y2025/V51/I10/270

Figures/Tables 13

Fig.1 Overall schematic diagram of joint learning

Fig.2 Schematic diagram of segmentation network

Fig.3 Eight subbands and their energy image after 3D discrete wavelet decomposition of floating image and fixed image

Fig.4 Structure of registration network Reg-MWNet

Fig.5 Schematic diagram of registration sub-network

Fig.6 Visualization results among different registration methods in the test dataset

Fig.7 Visualization results of registration networks with different layers

References 42

1	SIEGEL R, MILLER K, FUCHS H, et al. Cancer statistics, 2022. CA. 506. Cancer Journal of Clinic, 2022, 72(7-33): 507.
2	何小双, 徐丽娜, 刘冬, 等. 基于年龄-时期-队列模型的2005-2016年中国老年人恶性肿瘤流行特征分析. 中华肿瘤防治杂志, 2023, 30(11): 631- 638.
	HE X S, XU L N, LIU D, et al. Epidemiological characteristics of malignant tumors in the elderly in China from 2005 to 2016 based on age period cohort model. Chinese Journal of Cancer Prevention and Treatment, 2023, 30(11): 631- 638.
3	SUNG H, FERLAY J, SIEGEL R L, et al. Global cancer statistics 2020:GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: a Cancer Journal for Clinicians, 2018, 68(6): 394- 424. doi: 10.3322/caac.21492
4	傅强, 韩邦旻, 刘振湘, 等. 前列腺穿刺活检专家共识. 中华男科学杂志, 2022, 28(5): 462- 470.
	FU Q, HAN B M, LIU Z X, et al. Expert consensus on prostate biopsy. National Journal of Andrology, 2022, 28(5): 462- 470.
5	谭振霖, 郭圣文. 基于关键点的超声图像与磁共振图像多分辨率离散优化配准方法. 生物医学工程学杂志, 2023, 40(2): 202-207, 216.
	TAN Z L, GUO S W. Multiresolution discrete optimization registration method of ultrasound and magnetic resonance images based on key points. Journal of Biomedical Engineering, 2023, 40(2): 202-207, 216.
6	JACEWICZ M, GVNZEL K, RUD E, et al. Multicenter transperineal MRI-TRUS fusion guided outpatient clinic prostate biopsies under local anesthesia. Urologic Oncology: Seminars and Original Investigations, 2021, 39(7): 1- 7.
7	杨勇军, 贺显雅, 曾一鸣, 等. 电磁针尖引导局部麻醉经会阴前列腺多模态影像融合靶向穿刺学习曲线及经验体会. 临床泌尿外科杂志, 2024, 39(3): 235- 241.
	YANG Y J, HE X Y, ZENG Y M, et al. Learning curve and experience of multi-mode image fusion targeted transperineal biopsy technique using electromagnetic needle tracking under local anaesthesia. Journal of Clinical Urology, 2024, 39(3): 235- 241.
8	FU Y B, LEI Y, WANG T H, et al. Deep learning in medical image registration: a review. Physics in Medicine and Biology, 2020, 65(20): 1- 27.
9	KLEIN S, STARING M, MURPHY K, et al. elastix: a toolbox for intensity-based medical image registration. IEEE Transactions on Medical Imaging, 2010, 29(1): 196- 205. doi: 10.1109/TMI.2009.2035616
10	SHEN D G. Image registration by local histogram matching. Pattern Recognition, 2007, 40(4): 1161- 1172. doi: 10.1016/j.patcog.2006.08.012
11	YANG D S, BRAME S, El NAQA I, et al. Technical note: DIRART-a software suite for deformable image registration and adaptive radiotherapy research[EB/OL]. [2023-11-20]. https://europepmc.org/backend/ptpmcrender.fcgi?accid=PMC3017581&blobtype=pdf.
12	杭昊, 黄影平, 张栩瑞, 等. 面向道路场景语义分割的移动窗口变换神经网络设计. 光电工程, 2024, 51(1): 230304.
	HANG H, HUANG Y P, ZHANG X R, et al. Design of Swin Transformer for semantic segmentation of road scenes. Opto-Electronic Engineering, 2024, 51(1): 230304.
13	FAN J F, CAO X H, YAP P T, et al. BIRNet: brain image registration using dual-supervised fully convolutional networks. Medical Image Analysis, 2019, 54, 193- 206. doi: 10.1016/j.media.2019.03.006
14	BALAKRISHNAN G, ZHAO A, SABUNCU M R, et al. VoxelMorph: a learning framework for deformable medical image registration. IEEE Transactions on Medical Imaging, 2019, 38(8): 1788- 1800. doi: 10.1109/TMI.2019.2897538
15	ONOFREY J A, OKSUZ I, SARKAR S, et al. MRI-TRUS image synthesis with application to image-guided prostate intervention[C]//Proceedings of the International Workshop on Simulation and Synthesis in Medical Imaging. Berlin, Germany: Springer, 2016: 157-166.
16	YAN P, XU S, RASTINEHAD A R, et al. Adversarial image registration with application for MR and TRUS image fusion[C]//Proceedings of the Machine Learning in Medical Imaging: 9th International Workshop. Berlin, Germany: Springer, 2018: 197-204.
17	SHAO W, BANH L, KUNDER C A, et al. ProsRegNet: a deep learning framework for registration of MRI and histopathology images of the prostate. Medical Image Analysis, 2021, 68, 1019- 1030.
18	HASKINS G, KRUECKER J, KRUGER U, et al. Learning deep similarity metric for 3D MR-TRUS image registration[EB/OL]. [2023-11-20]. https://arxiv.org/pdf/1806.04548.
19	GUO H T, KRUGER M, XU S, et al. Deep adaptive registration of multi-modal prostate images. Computerized Medical Imaging and Graphics, 2020, 84, 101769. doi: 10.1016/j.compmedimag.2020.101769
20	SONG X, GUO H, XU X, et al. Cross-modal attention for MRI and ultrasound volume registration[C]//Proceedings of the Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2021: 66-75.
21	HU Y P, MODAT M, GIBSON E, et al. Label-driven weakly-supervised learning for multimodal deformable image registration[C]//Proceedings of the 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). Washington D. C., USA: IEEE Press, 2018: 1070-1074.
22	XU Z L, NIETHAMMER M. DeepAtlas: joint semi-supervised learning of image registration and segmentation[C]//Proceedings of the Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2019: 420-429.
23	SHAO W, BHATTACHARYA I, SOERENSEN S J, et al. Weakly supervised registration of prostate MRI and histopathology images[C]//Proceedings of the Medical Image Computing and Computer Assisted Intervention. Berlin, Germany: Springer, 2021: 98-107.
24	ZHANG D W, ZHENG Z L. Joint representation learning with deep quadruplet network for real-time visual tracking[C]//Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN). [S. l. ]: AAAI Press, 2020: 1-8.
25	ELMAHDY M S, WOLTERINK J M, SOKOOTI H, et al. Adversarial optimization for joint registration and segmentation in prostate CT radiotherapy[EB/OL]. [2023-11-20]. https://arxiv.org/pdf/1906.12223.
26	LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2017: 2117-2125.
27	MOK T C, CHUNG A C. Large deformation diffeomorphic image registration with laplacian pyramid networks[EB/OL]. [2023-11-20]. https://arxiv.org/pdf/2006.16148.
28	BIAN P C, ZHENG Z L, ZHANG D W. Light-weight multi-channel aggregation network for image super-resolution[C]//Proceedings of the Pattern Recognition and Computer Vision. Berlin, Germany: Springer, 2021: 287-297.
29	ISENSEE F, JAEGER P F, KOHL S A, et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 2021, 18(2): 203- 211. doi: 10.1038/s41592-020-01008-z
30	ROY A G, NAVAB N, WACHINGER C. Concurrent spatial and channel 'squeeze & excitation'in fully convolutional networks[EB/OL]. [2023-11-20]. https://arxiv.org/pdf/1803.02579.
31	LUO C C, LI Y M, LIN K M, et al. Wavelet synthesis Net for disparity estimation to synthesize DSLR calibre Bokeh effect on smartphones[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2020: 2407-2415.
32	GUARIGLIA E, SILVESTROV S. Fractional-wavelet analysis of positive definite distributions and wavelets on D'(C) D'(C)[C]//Proceedings of the Engineering mathematics Ⅱ: Algebraic, Stochastic and Analysis Structures for Networks, Data Classification and Optimization. Berlin, Germany: Springer, 2016: 337-353.
33	LIU P J, ZHANG H Z, ZHANG K, et al. Multi-level wavelet-CNN for image restoration[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington D.C., USA: IEEE Press, 2018: 773-782.
34	QU L Q, ZHANG Y Q, WANG S, et al. Synthesized 7T MRI from 3T MRI via deep learning in spatial and wavelet domains. Medical Image Analysis, 2020, 62, 101663. doi: 10.1016/j.media.2020.101663
35	GHASEMZADEH A, DEMIREL H. 3D discrete wavelet transform-based feature extraction for hyperspectral face recognition. IET Biometrics, 2018, 7(1): 49- 55. doi: 10.1049/iet-bmt.2017.0082
36	LI S T, PENG J L, KWOK J T, et al. Multimodal registration using the discrete wavelet frame transform[C]//Proceedings of the 18th International Conference on Pattern Recognition. Washington D.C., USA: IEEE Press, 2006: 877-880.
37	The Cancer Imaging Archive. Prostate MRI-TRUS biopsy dataset is available[EB/OL]. [2023-11-20]. https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=68550661.
38	KANG M, HU X J, HUANG W L, et al. Dual-stream pyramid registration network. Medical Image Analysis, 2022, 78, 102379. doi: 10.1016/j.media.2022.102379
39	CHEN J Y, HE Y F, FREY E C, et al. ViT-V-Net: vision transformer for unsupervised volumetric medical image registration[EB/OL]. [2023-11-20]. https://arxiv.org/pdf/2104.06468.
40	CHE T, WANG X, ZHAO K, et al. AMNet: adaptive multi-level network for deformable registration of 3D brain MR images[EB/OL]. [2023-11-20]. https://arxiv.org/pdf/2104.06468.
41	CHEN J Y, FREY E C, HE Y F, et al. TransMorph: Transformer for unsupervised medical image registration. Medical image analysis, 2022, 82, 102615. doi: 10.1016/j.media.2022.102615
42	GUO M H, LIU Z N, MU T J, et al. Beyond self-attention: external attention using two linear layers for visual tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(5): 5436- 5447.

[1]	LIU Zhaowei, FANG Yanhong, ZHENG Mingyu, SUO Bin. Lung Disease Diagnosis Method Based on Attention Mechanism and Multi-tasking [J]. Computer Engineering, 2025, 51(1): 332-342.
[2]	YOU Leqi, PEI Zhongmin, LUO Zhangkai. Ontology-Instances Joint Learning Method Fusing Entity Type Information [J]. Computer Engineering, 2022, 48(7): 82-88.
[3]	TAO Yang, BAO Linglang, HU Hao. Dimensionality Reduction Method Combining Representation Learning and Embedded Subspace Learning [J]. Computer Engineering, 2021, 47(6): 83-87,97.
[4]	DONG Yachao, LIU Hongzhe, XU Cheng. Person Re-Identification Method Based on Joint Fusion of Saliency Multi-Scale Features [J]. Computer Engineering, 2021, 47(6): 234-244,252.
[5]	XU Longzhuang, PENG Li, ZHU Fengzeng. Pedestrian Re-Identification Method Based on Multi-Task Pyramid Overlapping Matching [J]. Computer Engineering, 2021, 47(1): 239-245,254.

Please choose a citation manager

Content to export