MTM3D: 3D Medical Image Analysis Model Integrating Mamba and Improved TTM

doi:10.19678/j.issn.1000-3428.0070764

Abstract

Abstract:

Biomedical imaging is crucial in the diagnosis and treatment of various diseases. The application of deep learning methods to medical image analysis can enhance the readability of medical images and provide more reliable support for clinical decision-making. However, traditional medical image processing methods are limited in effectively capturing spatial features and complex structural information in 3D images, especially when handling complex 3D medical images generated by different imaging modalities. This often challenges the accuracy and generalization ability of the model. To address this challenge, an MTM3D model is proposed for medical image classification tasks. This model combines the excellent performance of the Mamba model in complex sequential tasks with the external memory storage function of the improved Token Turning Machine (TTM) network. By introducing a cyclic chain storage structure, MTM3D enables effective interaction of features from different spatial structures within memory units, thus enhancing its ability to capture complex spatial relationships. Furthermore, the incorporation of Mamba further strengthens the interaction between the memory and processing units, strengthening the generalization capability of the model and enhancing its performance across different medical imaging datasets. Experimental results on the MedMNIST v2 dataset demonstrate that MTM3D exhibits outstanding capabilities in understanding medical images. Compared with the current best medical image analysis networks, MTM3D improves the average Accuracy (ACC) by 3.97% and the average Area Under the Curve (AUC) by 2.00%, fully showcasing its tremendous potential in interpreting medical images and assisting healthcare professionals in diagnosis and treatment planning.

Key words: 3D medical image classification, Mamba, Token Turing Machine (TTM), Transformer, memory storage structure

摘要：

生物医学成像在诊断和治疗多种疾病中起着至关重要的作用。将深度学习方法应用于医学图像分析能够提高医学图像的可读性, 为临床决策提供更可靠的支持。然而, 传统的医学图像处理方法在有效捕获三维图像中的空间特征和复杂结构信息方面存在一定局限性, 尤其是在处理不同成像方式生成的复杂三维医学图像时, 模型的精度和泛化能力常常受限。针对这一挑战, 提出了一种MTM3D模型用于医学图像分类任务, 该模型结合了Mamba模型在复杂序列任务上的优异性能与改进令牌图灵机(TTM)网络的外部记忆存储功能。通过引入链式循环存储结构, MTM3D能够在记忆单元中有效交互不同空间结构的特征, 从而提升对复杂空间关系的捕捉能力; 此外, Mamba的引入进一步增强了记忆单元与处理单元的交互能力, 使模型具备更强的泛化能力, 在不同的医学图像数据集上表现出色。实验结果表明, MTM3D在MedMNIST v2数据集上的医学图像理解能力表现优异。相比现有最佳的医学图像分析网络, MTM3D的平均准确率(ACC)提升了3.97%, 平均曲线下面积(AUC)提升了2.00%, 充分展示了其在医学图像解读和协助医疗专业人员进行诊断与治疗规划中的巨大潜力。

关键词: 三维医学图像分类, Mamba, 令牌图灵机, Transformer, 记忆存储结构

YANG Yang, WEI Hongkai, SUN Shijie, HU Hongli, WANG Rong, WANG Tiantian. MTM3D: 3D Medical Image Analysis Model Integrating Mamba and Improved TTM[J]. Computer Engineering, 2026, 52(9): 193-203.

杨洋, 魏弘凯, 孙士杰, 胡红利, 王荣, 王天添. MTM3D: 融合Mamba与改进TTM的三维医学图像分析模型[J]. 计算机工程, 2026, 52(9): 193-203.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0070764

https://www.ecice06.com/EN/Y2026/V52/I9/193

Figures/Tables 13

Fig.1 MTM3D model framework

Fig.2 Comparison of memory structures between TTM and MTM3D models

Fig.3 Five types of memory-enhanced units in MTM3D

Fig.4 Visualization of the MedMNIST v2 dataset

Fig.5 Visualization of inference results

References 36

1	WEBB A. Introduction to biomedical imaging[M]. [S. l. ]: John Wiley & Sons, 2022.
2	DOU Q , CHEN H , YU L Q , et al. Automatic detection of cerebral microbleeds from MR images via 3D convolutional neural networks. IEEE Transactions on Medical Imaging, 2016, 35 (5): 1182- 1195. doi: 10.1109/TMI.2016.2528129
3	RASAL R, CASTRO D C, PAWLOWSKI N, et al. Deep structural causal shape models[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2023: 400-432.
4	LIU F Z, CAI J Z, HUO Y K, et al. JSSR: a joint synthesis, segmentation, and registration system for 3D multi-modal image alignment of large-scale pathological CT scans[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 257-274.
5	李翠云, 白静, 郑凉. 融合边缘增强注意力机制和U-Net网络的医学图像分割. 图学学报, 2022, 43 (2): 273- 278.
	LI C Y , BAI J , ZHENG L . A U-Net based contour enhanced attention for medical image segmentation. Journal of Graphics, 2022, 43 (2): 273- 278.
6	张淑军, 彭中, 李辉. SAU-Net: 基于U-Net和自注意力机制的医学图像分割方法. 电子学报, 2022, 50 (10): 2433- 2442.
	ZHANG S J , PENG Z , LI H . SAU-Net: medical image segmentation method based on U-Net and self-attention. Acta Electronica Sinica, 2022, 50 (10): 2433- 2442.
7	黄帅帅, 张清华, 靳朋仁, 等. 基于权重生成图卷积神经网络的链接预测方法. 重庆邮电大学学报(自然科学版), 2025, 37 (3): 375- 386.
	HUANG S S , ZHANG Q H , JIN P R , et al. Link prediction method based on weighted-generator graph convolutional networks. Journal of Chongqing University of Posts and Telecommunications (Natural Science Edition), 2025, 37 (3): 375- 386.
8	谢娟英, 张凯云. SOSNet: 一种非对称编码器-解码器结构的非小细胞肺癌CT图像分割模型. 电子学报, 2024, 52 (3): 824- 837.
	XIE J Y , ZHANG K Y . SOSNet: an asymmetric encoder-decoder structure model for automatic segmenting non-small cell lung cancer CT images. Acta Electronica Sinica, 2024, 52 (3): 824- 837.
9	RAZZAK M I, NAZ S, ZAIB A. Deep learning for medical image processing: overview, challenges and the future[M]//DEY N, ASHOUR A S, BORRA S, et al. Classification in BioApps: automation of decision making. Berlin, Germany: Springer, 2018: 323-350.
10	王欣雨, 刘慧, 朱积成, 等. 基于高低频特征分解的深度多模态医学图像融合网络. 图学学报, 2024, 45 (1): 65- 77.
	WANG X Y , LIU H , ZHU J C , et al. Deep multimodal medical image fusion network based on high-low frequency feature decomposition. Journal of Graphics, 2024, 45 (1): 65- 77.
11	SAHA A, TUSHAR F I, FARYNA K, et al. Weakly supervised 3D classification of chest CT using aggregated multi-resolution deep segmentation features[C]//Proceedings of the Medical Imaging 2020: Computer-Aided Diagnosis. Houston, USA: SPIE, 2020: 39-44.
12	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 6000-6010.
13	GRAVES A, WAYNE G, DANIHELKA I. Neural turing machines[EB/OL]. [2024-10-12]. https://arxiv.org/abs/1410.5401.
14	RYOO M S, GOPALAKRISHNAN K, KAHATAPITIYA K, et al. Token Turing Machines[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2023: 19070-19081.
15	GU A, DAO T. Mamba: linear-time sequence modeling with selective state spaces[EB/OL]. [2024-10-12]. https://arxiv.org/abs/2312.00752.
16	YANG J C , SHI R , WEI D L , et al. MedMNIST v2—a large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific Data, 2023, 10 (1): 41. doi: 10.1038/s41597-022-01721-8
17	BORE J C , LI P Y , JIANG L , et al. A long short-term memory network for sparse spatiotemporal EEG source imaging. IEEE Transactions on Medical Imaging, 2021, 40 (12): 3787- 3800. doi: 10.1109/TMI.2021.3097758
18	CHEN X , LOWERISON M R , DONG Z J , et al. Localization free super-resolution microbubble velocimetry using a long short-term memory neural network. IEEE Transactions on Medical Imaging, 2023, 42 (8): 2374- 2385. doi: 10.1109/TMI.2023.3251197
19	ZHANG R , QIN B , ZHAO J , et al. Locating X-ray coronary angiogram keyframes via long short-term spatiotemporal attention with image-to-patch contrastive learning. IEEE Transactions on Medical Imaging, 2024, 43 (1): 51- 63. doi: 10.1109/TMI.2023.3286859
20	CHEN Y R , ZHANG H , WANG Y N , et al. MAMA net: multi-scale attention memory autoencoder network for anomaly detection. IEEE Transactions on Medical Imaging, 2021, 40 (3): 1032- 1041. doi: 10.1109/TMI.2020.3045295
21	WANG P Y , ZHANG H Q , ZHU M L , et al. MGIML: cancer grading with incomplete radiology-pathology data via memory learning and gradient homogenization. IEEE Transactions on Medical Imaging, 2024, 43 (6): 2113- 2124. doi: 10.1109/TMI.2024.3355142
22	RYOO M S, PIERGIOVANNI A, ARNAB A, et al. TokenLearner: what can 8 learned tokens do for images and videos?[EB/OL]. [2024-10-12]. https://arxiv.org/abs/2106.11297.
23	JAEGLE A, GIMENO F, BROCK A, et al. Perceiver: general perception with iterative attention[C]//Proceedings of International Conference on Machine Learning. [S. l. ]: PMLR, 2021: 4651-4664.
24	AHMADI N , TSANG M Y , GU A N , et al. Transformer-based spatio-temporal analysis for classification of aortic stenosis severity from echocardiography cine series. IEEE Transactions on Medical Imaging, 2023, 43 (1): 366- 376.
25	YANG Z Y , PAN J J , DAI J , et al. Self-supervised lightweight depth estimation in endoscopy combining CNN and transformer. IEEE Transactions on Medical Imaging, 2024, 43 (5): 1934- 1944. doi: 10.1109/TMI.2024.3352390
26	WU R K, LIU Y H, LIANG P C, et al. UltraLight VM-UNet: parallel vision mamba significantly reduces parameters for skin lesion segmentation[EB/OL]. [2024-10-12]. https://arxiv.org/abs/2403.20035.
27	RUAN J, XIANG S. VM-UNet: vision Mamba UNet for medical image segmentation[EB/OL]. [2024-10-12]. https://arxiv.org/abs/2402.02491.
28	ZHOU J Y, JIANG M Z, WU J D, et al. MGI: multimodal contrastive pre-training of genomic and medical imaging[EB/OL]. [2024-10-12]. https://arxiv.org/abs/2406.00631.
29	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, USA: IEEE Press, 2016: 770-778.
30	FEURER M, KLEIN A, EGGENSPERGER K, et al. Efficient and robust automated machine learning[C]//Proceedings of NeurIPS 2015. [S. l. ]: Curran Associates, Inc., 2015: 1-9.
31	JIN H F, SONG Q Q, HU X. Auto-keras: an efficient neural architecture search system[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, USA: ACM Press, 2019: 1946-1956.
32	LIU J W, LI Y, CAO G T, et al. Feature pyramid vision transformer for MedMNIST classification decathlon[C]//Proceedings of the International Joint Conference on Neural Networks (IJCNN). Padua, Italy: IEEE Press, 2022: 1-8.
33	YANG J C , HUANG X Y , HE Y , et al. Reinventing 2D convolutions for 3D images. IEEE Journal of Biomedical and Health Informatics, 2021, 25 (8): 3009- 3018. doi: 10.1109/JBHI.2021.3049452
34	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[EB/OL]. [2024-10-12]. https://arxiv.org/abs/2010.11929.
35	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of International Conference on Machine Learning. [S. l. ]: PMLR, 2021: 8748-8763.
36	LIU Z, LIN Y T, CAO Y, et al. Swin Transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, Canada: IEEE Press, 2022: 9992-10002.

[1]	ZHANG Ruijia, MA Huifang, ZHANG Yingyue, PENG Shengjiang. Fault Detection in Oil-Immersed Power Transformers Based on Generalizable Graph Knowledge Distillation [J]. Computer Engineering, 2026, 52(9): 449-456.
[2]	GAO Liulong, HUANG Zhengkun, JIANG Xiaowei, SUN Gongxing, LI Jiafeng. Survey of Deep Learning Models in High Energy Physics Jet Tagging [J]. Computer Engineering, 2026, 52(8): 101-113.
[3]	JIN Jing, HU Chudi, CHEN Gang. Medical Image Segmentation Method Based on Multi-Scale Self-Attention Transformer [J]. Computer Engineering, 2026, 52(7): 199-209.
[4]	WANG Xiaochuan, LIU Qingqing, WANG Chen, MU Hengchen, LIAO Yaoming, LIU Ruijun. GeoPriFormer: A Precipitation Spatial Interpolation Method with Geographic Priors Embedding [J]. Computer Engineering, 2026, 52(7): 179-188.
[5]	HUO Jiuyuan, LI Xin, CHANG Chen, ZHANG Yaonan. Dual-Channel Rolling Bearing Fault Diagnosis Method Based on ACNN-LFSwin Transformer [J]. Computer Engineering, 2026, 52(5): 430-444.
[6]	TIAN Hui, DUAN Xinlong, HAO Qiya, SUI Wenhao, MA Yuying, YU Zuhua, XU Yang, CAO Yangjie. Cell Counting Method Combining Multi-Scale Feature Fusion and Improved ViT [J]. Computer Engineering, 2026, 52(5): 203-215.
[7]	SHEN Danyang, MAI Wen. Automatic Modulation Recognition of Communication Signals Based on the ResNet-Transformer [J]. Computer Engineering, 2026, 52(5): 383-395.
[8]	LI Yiyang, LU Shenglian, WANG Jijie, CHEN Ming. Review of DETR Object Detection Algorithm Based on Transformer [J]. Computer Engineering, 2026, 52(4): 62-81.
[9]	WANG Shasha, LI Weitao, LIU Xingyu, GAO Hui. Domain Adaptive Remote Sensing Image Segmentation Based on Hierarchical Attention [J]. Computer Engineering, 2026, 52(4): 176-186.
[10]	LI Bin, GUO Yi. Deep Reinforcement Learning Algorithms for Heterogeneous Multiple Knapsack Problems [J]. Computer Engineering, 2026, 52(4): 140-162.
[11]	LIU Ze, SONG Tinglun, SHI Xianrang, SU Yang, ZHAO Qun. Research on Pavement Anomaly Detection Technology Based on Improved RT-DETR [J]. Computer Engineering, 2026, 52(4): 187-199.
[12]	SONG Chaoqi, LIU Ying, HE Jinglu, LI Daxiang. Few-shot Image Classification Method Based on Salient Position Interaction Transformer [J]. Computer Engineering, 2026, 52(2): 167-176.
[13]	SONG Quanzhen, CHEN Zuojun, QIN Pinle, ZENG Jianchao. Superpixel Guide for Transformer Low-Light Image Denoising Method [J]. Computer Engineering, 2026, 52(2): 186-196.
[14]	CHEN Dongji, LAI Huicheng, GAO Guxue, MA Jun, LI Junkai, QUAN Hutuo. Knowledge Distillation-based Transformer for Human-Object Interaction Detection [J]. Computer Engineering, 2026, 52(1): 206-216.
[15]	LI Bingye, BAO Yu, CAO Wei, LI Mingze, XU Fangzheng. Cross-layer Channel Pruning Method Based on Variable Sequences [J]. Computer Engineering, 2026, 52(1): 381-389.

Please choose a citation manager

Content to export