基于多模态融合的宫颈上皮内瘤变辅助诊断

doi:10.19678/j.issn.1000-3428.0069544

摘要/Abstract

摘要：

近年来, 深度学习在医学图像处理领域的应用取得了显著进展。然而, 现有的方法大多采用图像分类的方式来辅助医生进行病理诊断, 在某些情况下可能会出现缺乏可解释性的问题。为了解决这一问题, 提出一种基于多模态特征融合的宫颈上皮内瘤变(CIN)分类模型。在这个多模态分类模型中, 患者的病理图像作为图像模态数据, 而相应的病理报告描述则作为文本模态数据。当前多模态模型在处理不同模态信息时, 大多采用简单拼接的方法组合各个模态的特征, 往往忽略了跨模态信息之间的联系。为了促进跨模态信息之间的交互, 采用一种改进的Nonlocal注意力机制, 以提高模型对数据的全面理解能力。同时, 考虑到现实中并非每张图像都有相应的病理报告, 因此在训练中采用一种文本"丢弃"的编码策略, 以确保训练好的模型在文本信息缺失时仍能够实现较高的分类准确率。实验结果表明, 该模型在CIN分类方面具有良好的性能, 相较于传统的ResNet、ConvNeXt、MobileViT等单模态方法在分类准确率方面提升了7~9百分点。

关键词: 多模态融合, 注意力机制, 病理辅助诊断, 深度学习, 宫颈上皮内瘤变

Abstract:

In recent years, deep learning has made significant progress in the field of medical image processing. Most existing methods use image classification to assist physicians in pathological diagnosis, which may suffer from a lack of interpretability. To address this problem, this study proposes a multimodal feature fusion-based classification model for Cervical Intraepithelial Neoplasia (CIN). In this multimodal classification model, a patient's pathology image is used as the image modality data, whereas the corresponding pathology report description is used as the text modality data. Most current multimodal models use a simple splicing method to combine the features of each modality when dealing with different modal information. However, this method often ignores the connection between cross-modal information. To facilitate the interaction between cross-modal information, this study adopts an improved nonlocal attention mechanism to enhance the model's ability to understand the data comprehensively. Meanwhile, considering that not every image has a corresponding pathology report, the study leverages a text-discarding coding strategy during training to ensure that the trained model can achieve high classification accuracy despite missing text information. Experimental results show that the model performs well in CIN classification, and the classification accuracy is improved by 7—9 percentage points compared to those of traditional unimodal methods such as ResNet, ConvNeXt, and MobileViT.

Key words: multimodal fusion, attention mechanism, pathology assisted diagnosis, deep learning, Cervical Intraepithelial Neoplasia (CIN)

冯赛赛, 葛东峰, 李涛, 刘一靖, 冀治航, 王琳, 张明川. 基于多模态融合的宫颈上皮内瘤变辅助诊断[J]. 计算机工程, 2025, 51(12): 304-310.

FENG Saisai, GE Dongfeng, LI Tao, LIU Yijing, JI Zhihang, WANG Lin, ZHANG Mingchuan. Assisted Diagnosis of Cervical Intraepithelial Neoplasia Based on Multimodal Fusion[J]. Computer Engineering, 2025, 51(12): 304-310.

https://www.ecice06.com/CN/Y2025/V51/I12/304

图/表 9

图1 模型主体框架图

Fig.1 Main frame diagram of the model

图2 CINDR数据集示例

Fig.2 CINDR dataset sample

图3 文本编码策略

Fig.3 Text encoding strategy

图4 不同丢弃率的准确性对比

Fig.4 Comparison of accuracy with different discarding rates

图5 表征向量的t-SNE降维可视化

Fig.5 The t-SNE dimensionality reduction visualization of representation vectors

图6 模型对于图像中关注区域的可视化展示

Fig.6 Visualization of the model′s concerned region in the image

参考文献 26

1	ØVESTAD I T , ENGESÆTER B , HALLE M K , et al. High-grade cervical intraepithelial neoplasia (CIN) associates with increased proliferation and attenuated immune signaling. International Journal of Molecular Sciences, 2022, 23 (1): 373.
2	BA W , WANG S , SHANG M , et al. Assessment of deep learning assistance for the pathological diagnosis of gastric cancer. Modern Pathology, 2022, 35 (9): 1262- 1268. doi: 10.1038/s41379-022-01073-z
3	李建威, 吕晓琪, 谷宇. 基于改进ConvNeXt的皮肤镜图像分类方法. 计算机工程, 2023, 49 (10): 239-246, 254. doi: 10.19678/j.issn.1000-3428.0066050
	LI J W , LV X Q , GU Y . Dermoscopy image classification method based on improved ConvNeXt. Computer Engineering, 2023, 49 (10): 239-246, 254. doi: 10.19678/j.issn.1000-3428.0066050
4	张雪芹, 李天任. 基于Cycle-GAN和改进DPN网络的乳腺癌病理图像分类. 浙江大学学报(工学版), 2022, 56 (4): 727- 735.
	ZHANG X Q , LI T R . Classification of breast cancer pathological images based on Cycle-GAN and improved DPN network. Journal of Zhejiang University (Engineering Science), 2022, 56 (4): 727- 735.
5	XUE D , ZHOU X , LI C , et al. An application of transfer learning and ensemble learning techniques for cervical histopathology image classification. IEEE Access, 2020, 8, 104603- 104618. doi: 10.1109/ACCESS.2020.2999816
6	WANG Y, BEUVING F, NONNEKES J, et al. Freezing of gait detection in Parkinson's disease via multimodal analysis of EEG and accelerometer signals[C]//Proceedings of 202042nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society. Washington D.C., USA: IEEE Press, 2020: 847-850.
7	LI Y , MAMOUEI M , SALIMI-KHORSHIDI G , et al. Hi-BEHRT: hierarchical transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records. IEEE Journal of Biomedical and Health Informatics, 2022, 27 (2): 1106- 1117.
8	LIU F, WU X, GE S, et al. Exploring and distilling posterior and prior knowledge for radiology report generation[C]//Proceedings of 2021 IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2021: 13748-13757.
9	张炯, 王丽芳, 蔺素珍, 等. 局部全局特征耦合与交叉尺度注意的医学图像融合. 计算机工程, 2023, 49 (3): 238- 247. doi: 10.19678/j.issn.1000-3428.0064891
	ZHANG J , WANG L F , LIN S Z , et al. Medical image fusion with local-global feature coupling and cross-scale attention. Computer Engineering, 2023, 49 (3): 238- 247. doi: 10.19678/j.issn.1000-3428.0064891
10	HAN J , JIANG G , OUYANG G , et al. A multimodal approach for identifying autism spectrum disorders in children. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2022, 30, 2003- 2011.
11	SHI Z , ZHANG C , YE D , et al. MMI-Fuse: multimodal brain image fusion with multiattention module. IEEE Access, 2022, 10, 37200- 37214. doi: 10.1109/ACCESS.2022.3163260
12	LIU T , HUANG J , LIAO T , et al. A hybrid deep learning model for predicting molecular subtypes of human breast cancer using multimodal data. IRBM, 2022, 43 (1): 62- 74.
13	WU Y, MA J, HUANG X, et al. DeepMMSA: a novel multimodal deep learning method for non-small cell lung cancer survival analysis[C]//Proceedings of 2021 IEEE International Conference on Systems, Man, and Cybernetics. Washington D.C., USA: IEEE Press, 2021: 1468-1472.
14	WANG R, YAO Q, LAI H, et al. ECAMP: entity-centered context-aware medical vision language pre-training[EB/OL]. [2024-02-12]. https://arxiv.org/abs/2312.13316.
15	WANG X, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2018: 7794-7803.
16	LIU Z, LV Q, LI Y, et al. MedAugment: universal automatic data augmentation plug-in for medical image analysis[EB/OL]. [2024-02-12]. https://arxiv.org/abs/2306.17466.
17	HENDRYCKS D, MU N, CUBUK E D, et al. AugMix: a simple data processing method to improve robustness and uncertainty[EB/OL]. [2024-02-12]. https://arxiv.org/abs/1912.02781.
18	HUANG Z , ZHU X , DING M , et al. Medical image classification using a light-weighted hybrid neural network based on PCANet and DenseNet. IEEE Access, 2020, 8, 24697- 24712. doi: 10.1109/ACCESS.2020.2971225
19	WOO S, DEBNATH S, HU R, et al. ConvNeXt v2: co-designing and scaling convnets with masked autoencoders[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2023: 16133-16142.
20	HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2016: 770-778.
21	MAAZ M, SHAKER A, CHOLAKKAL H, et al. EdgeNeXt: efficiently amalgamated CNN-transformer architecture for mobile vision applications[C]//Proceedings of European Conference on Computer Vision. Cham, Switzerland: Springer Nature, 2022: 3-20.
22	MEHTA S, RASTEGARI M. MobileViT: light-weight, general-purpose, and mobile-friendly vision transformer[EB/OL]. [2024-02-12]. https://arxiv.org/abs/2110.02178.
23	ZHANG Z, XIE Y, XING F, et al. MDNet: a semantically and visually interpretable medical image diagnosis network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2017: 6428-6436.
24	ZHANG Z , CHEN P , SHI X , et al. Text-guided neural network training for image recognition in natural scenes and medicine. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43 (5): 1733- 1745.
25	VINYALS O, TOSHEV A, BENGIO S, et al. Show and tell: a neural image caption generator[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2015: 3156-3164.
26	VAN DER MAATEN L , HINTON G . Visualizing data using t-SNE. Journal of Machine Learning Research, 2008, 9, 2579- 2605.

[1]	黄金贵, 刘朋, 唐文胜. MMD-YOLOv7:黑暗条件下车辆检测方法[J]. 计算机工程, 2025, 51(9): 340-349.
[2]	符家成, 田瑾, 张玉金, 方志军. 结合前置三元组集的知识图谱推荐[J]. 计算机工程, 2025, 51(9): 101-109.
[3]	周晨阳, 刘雪宇, 梁少华, 吴永飞. 基于Swin Transformer的肾动脉血管检测分割与定量分析[J]. 计算机工程, 2025, 51(9): 252-267.
[4]	翟志鹏, 曹阳, 沈琴琴, 施佺. 基于多时空图融合与动态注意力的交通流预测[J]. 计算机工程, 2025, 51(9): 139-148.
[5]	徐瀅, 傅紫薇, 张伟, 陈云芳. 基于抽象语法树嵌入的智能合约漏洞检测技术[J]. 计算机工程, 2025, 51(9): 149-157.
[6]	马淦, 谷雨, 彭冬亮. 结合改进YOLOv5s和动态数据增强的海面舰船检测[J]. 计算机工程, 2025, 51(9): 294-305.
[7]	陈彦如, 刘珂良, 冉茂亮. 基于深度强化学习的外卖即时配送实时优化[J]. 计算机工程, 2025, 51(9): 328-339.
[8]	林帆, 李建华. 基于多阶门控聚合网络的光学化学结构识别[J]. 计算机工程, 2025, 51(8): 364-372.
[9]	武东辉, 王金凤, 仇森, 刘国志. 基于EWBiLSTM-ATT的数据手套手语识别[J]. 计算机工程, 2025, 51(8): 107-119.
[10]	倪源松, 韩军, 邹小燕, 胡广怡, 王文帅. 两阶段自适应分块输电线路螺栓缺陷检测方法[J]. 计算机工程, 2025, 51(8): 281-291.
[11]	郝宏达, 罗健旭. 基于多尺度区域特征融合的多器官语义分割模型[J]. 计算机工程, 2025, 51(8): 270-280.
[12]	张昭理, 李家豪, 刘海, 石佛波, 何嘉文. 基于个性化遗忘建模的知识追踪方法[J]. 计算机工程, 2025, 51(8): 120-130.
[13]	闫建红, 刘芝妍, 王震. 融合时空注意力机制的多尺度卷积车辆轨迹预测[J]. 计算机工程, 2025, 51(8): 406-414.
[14]	武东辉, 王金凤, 仇森, 刘国志. 基于EWBiLSTM-ATT的数据手套手语识别[J]. 计算机工程, 2025, 51(8): 107-119.
[15]	李姜辛, 王鹏, 汪卫. 多机理指导的深度学习工业时序预测框架[J]. 计算机工程, 2025, 51(7): 47-58.

选择文件类型/文献管理软件名称

选择包含的内容