相似实例引导下融合异质图的医学影像报告生成

doi:10.19678/j.issn.1000-3428.0252152

摘要/Abstract

摘要： 医学影像报告自动生成任务存在影像对比度低、异常区域较小的问题，仅依靠影像信息难以精准刻画异常特征，因此如何引入外部知识来增强视觉表征成为解决问题的关键。此外，异常特征的共现关系复杂，依赖单一样本难以捕捉，如何利用相似实例建模共现模式至关重要。针对上述挑战，提出一种融合异质图的相似实例引导（SIG）方法，用于生成医学影像报告，包括结合异质图的图像特征记忆模块和相似实例特征融合（SIFF）模块。结合异质图的图像特征记忆模块提取报告实体关系，构建与报告对应的异质图为桥梁，引导模型关注图像异常区域，增强异常视觉特征；SIFF模块检索相似实例，融合相似实例的异常视觉特征，在增强异常区域特征表达的同时，获取更全面的异常信息。在 IU X-Ray 和 MIMIC-CXR 这两个医学影像数据集上验证所提方法的有效性，实验结果表明，所提方法在 BLEU 系列评分指标上表现优秀，在IU X-Ray数据集上BLEU-1~BLEU-4分别为 0.539、0.353、0.265、0.193。同时，该方法在 METEOR 和 ROUGE-L 指标上的表现同样卓越。所提方法在自然语言生成指标和生成报告的准确性、完整性方面优于现有主流方法。

关键词: 医学影像报告生成, 相似实例引导, 异质图, 特征融合, 图像-文本生成

Abstract: Medical report generation from images is challenging owing to the low image contrast and small size of abnormal regions, making it difficult to accurately capture abnormal features using visual information alone. Therefore, the introduction of external knowledge to enhance visual representations is a key issue. Additionally, the co-occurrence patterns of abnormal features are complex and cannot be effectively captured from a single instance, making it crucial to leverage similar cases to model such patterns. To address these challenges, a Similar Instance Guidance (SIG) approach that fuses heterogeneous graphs for medical image report generation is presented. This approach consists of two main components: an image feature memory module incorporating heterogeneous graphs and a Similar Instance Feature Fusion (SIFF) module. The image feature memory module incorporating heterogeneous graphs extracts entity relationships from reports and constructs a corresponding heterogeneous graph as a bridge, guiding the model's attention to the abnormal regions of the image, thus enhancing abnormal visual features. SIFF retrieves similar instances and integrates their abnormal visual features, thereby augmenting the representation of abnormal regions, while acquiring a more comprehensive understanding of the abnormal information. To verify the effectiveness of the proposed method, experiments are conducted on both IU X-Ray and MIMIC-CXR datasets. The results indicate that the proposed method outperforms baseline methods on BLEU-based metrics, yielding BLEU-1-BLEU-4 values of 0.539, 0.353, 0.265, and 0.193, respectively, on the IU X-Ray dataset. Additionally, it excels in the METEOR and ROUGE-L metrics, indicating that the proposed method outperforms existing methods in terms of natural language generation metrics as well as the accuracy and completeness of the generated reports.

Key words: Medical image Report Generation (MRG), Similar Instance Guidance (SIG), heterogeneous graph, feature fusion, image—text generation

中图分类号:

TP391

李俊亮, 马俊朋, 刘梦萱, 刘玉雪, 张俊三. 相似实例引导下融合异质图的医学影像报告生成[J]. 计算机工程, 2026, 52(11): 423-434.

LI Junliang, MA Junpeng, LIU Mengxuan, LIU Yuxue, ZHANG Junsan. Similar Instance Guidance Combining Heterogeneous Graph for Medical Image Report Generation[J]. Computer Engineering, 2026, 52(11): 423-434.

https://www.ecice06.com/CN/Y2026/V52/I11/423

参考文献

[1] VINYALS O, TOSHEV A, BENGIO S, et al. Show and tell: a neural image caption generator[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2015:3156-3164.
[2] JING B Y, XIE P T, XING E P. On the automatic generation of medical imaging reports[EB/OL].[2025-01-15]. https://arxiv.org/pdf/1711.08195.
[3] ZHANG Z Z, XIE Y P, XING F Y, et al. MDNet: a semantically and visually interpretable medical image diagnosis network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2017:6428-6436.
[4] SHIN H C, ROBERTS K, LU L, et al. Learning to read chest X-rays: recurrent neural cascade model for automated image annotation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2016:2497-2506.
[5] LIU F L, YIN C C, WU X, et al. Contrastive attention for automatic chest X-ray report generation[EB/OL].[2025-01-15]. https://arxiv.org/pdf/2106.06965.
[6] LI C Y, LIANG X D, HU Z T, et al. Hybrid retrieval-generation reinforced agent for medical image report generation[EB/OL].[2025-01-15]. https://arxiv.org/pdf/1805.08298.
[7] CHEN Z H, SONG Y, CHANG T-H, et al. Generating radiology reports via memory-driven transformer[EB/OL].[2025-01-15]. https://arxiv.org/pdf/2010.16056.
[8] CHEN Z H, SHEN Y L, SONG Y, et al. Cross-modal memory networks for radiology report generation[EB/OL].[2025-01-15]. https://arxiv.org/pdf/2204.13258.
[9] 邢素霞, 方俊泽,鞠子涵,等.基于记忆驱动的多模态医学影像报告自动生成研究[J]. 生物医学工程学杂志, 2024, 41(1): 60-69. XING S X, FANG J Z, JU Z H, et al. Research on automatic generation of multimodal medical image reports based on memory driven[J]. Journal of Biomedical Engineering, 2024, 41(1): 60-69. (in Chinese)
[10] 沈秀轩, 吴春雷,冯叶棋等.基于双分支特征融合的医学报告生成方法[J]. 计算机工程, 2023, 49(6): 274-283,291.SHEN X X,WU C L,FENG Y Q, et al. Medical report generation method based on dual-branch feature fusion[J]. Computer Engineering, 2023, 49(6): 274-283,291.(in Chinese)
[11] HOU W J, XU K S, CHENG Y, et al. ORGAN: observation-guided radiology report generation via tree reasoning[EB/OL].[2025-01-15]. https://arxiv.org/pdf/2306.06466.
[12] SONG X, ZHANG X, JI J, et al. Cross-modal contrastive attention model for medical report generation[C]//Proceedings of the 29th International Conference on Computational Linguistics. New York, USA: [s.n], 2022:2388-2397.
[13] LIU F L, WU X, GE S, et al. Exploring and distilling posterior and prior knowledge for radiology report generation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2021: 1113-1117.
[14] JIN H B, CHE H X, LIN Y, et al. PromptMRG: diagnosis-driven prompts for medical report generation[C]//Proceedings of the 38th AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024:2607-2615.
[15] KIRILLOV A, MINTUN E, RAVI N, et al. Segment anything[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2023: 4015-4026.
[16] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2016:770-778.
[17] JAIN S, AGRAWAL A, SAPORTA A, et al. RadGraph: extracting clinical entities and relations from radiology report[EB/OL].[2025-01-15]. https://arxiv.org/pdf/2106.14463.
[18] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL].[2025-01-15]. https://arxiv.org/abs/1706.03762?ref=andrealarosa.org.
[19] REINERS N, GUREVYCH I. Sentence-BERT: sentence embeddings using siamese BERT-networks[EB/OL].[2025-01-15]. https://arxiv.org/pdf/1908.10084.
[20] LI J, LI S B, HU Y, et al. A self-guided framework for radiology report generation[EB/OL].[2025-01-15]. https://arxiv.org/pdf/2206.09378.
[21] DEMNNER-FUSHMAN D, KOHLI M D, ROSENMAN M B, et al. Preparing a collection of radiology examinations for distribution and retrieval[J]. Journal of the American Medical Informatics Association, 2016, 23(2):304-310.
[22] JOHNSON A E W, POLLARD T J, GREENBAUM N R, et al. MIMIC-CXR: a large publicly available database of labeled chest radiographs[EB/OL].[2025-01-15]. https://arxiv.org/pdf/1901.07042.
[23] ROUKOS S, PAPINENI K, WARD T, et al. BLEU: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Philadelphia, USA: Association for Computational Linguistics, 2002:311-318.
[24] LIN C Y. ROUGE: a package for automatic evaluation of summaries[C]//Proceedings of Conference on Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, 2004:74-81.
[25] BANERJEE S, LAVIE A. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments[C]//Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics. Michigan, USA: Association for Computational Linguistics, 2005:65-72.
[26] DENG J, DONG W, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2009:248-255.
[27] LOSHCHILOV I, HUTTER F. Decoupled weight decay regularization[EB/OL].[2025-01-15]. https://arxiv.org/pdf/1711.05101.
[28] YANG Y, YU J, ZHANG J, et al. Joint embedding of deep visual and semantic features for medical image report generation[J]. IEEE Transactions on Multimedia, 2021, 25: 167-178.
[29] ZHANG J, SHEN X, WAN S, et al. A novel deep learning model for medical report generation by inter-intra information calibration[J]. IEEE Journal of Biomedical and Health Informatics, 2023, 27: 5110-5121.
[30] YANG S X, WU X, GE S, et al. Radiology report generation with a learned knowledge base and multi-modal alignment[J]. Medical Image Analysis, 2023, 86:102798.
[31] ZHANG K, JIANG H, ZHANG J, et al. Semi-supervised medical report generation via graph-guided hybrid feature consistency[J]. IEEE Transactions on Multimedia, 2023, 26: 904-915.
[32] WANG Z Y, LIU L Q, WANG L, et al. R2GenGPT: radiology report generation with frozen LLMs[J]. Meta-Radiology, 2023, 1(3): 100033.
[33] JIN Y, CHEN W, TIAN Y, et al. Improving radiology report generation with D2-Net: when diffusion meets discriminator[C]//Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D.C., USA: IEEE Press, 2024:2215-2219.
[34] LIU Z Z, ZHU Z F, ZHENG S, et al. From observation to concept: a flexible multi-view paradigm for medical report generation[J]. IEEE Transactions on Multimedia, 2024, 26: 5987-5995.
[35] CHEN W X, LIU Y, WANG C, et al. Cross-modal causal intervention for medical report generation[EB/OL].[2025-01-15]. https://arxiv.org/pdf/2303.09117.
[36] WANG X, WANG F L, WANG H W, et al. Activating associative disease-aware vision token memory for LLM-based x-ray report generation[EB/OL].[2025-01-15]. https://arxiv.org/pdf/2303.09117.
[37] TU T, AZIZI S, DRIESS D, et al. Towards generalist biomedical AI[J]. NEJM AI, 2024, 3(1): 102-105.

选择文件类型/文献管理软件名称

选择包含的内容