Abstract: Medical report generation from images is challenging due to low image contrast and the small size of abnormal regions, making it difficult to accurately capture abnormal features using visual information alone. Therefore, introducing external knowledge to enhance visual representation becomes a key issue. In addition, the co-occurrence patterns of abnormal features are complex and cannot be effectively captured from a single instance, making it crucial to leverage similar cases to model such patterns. To address the aforementioned challenges, a Similar-Instance Guided method for medical report generation is proposed, consisting of two main components: Image Feature Memory Module Incorporating Heterogeneous Graphs(FMHG) and Similar Instance Feature Fusion Module(SIFF). FMHG extracts entity relationships from the report and constructs a corresponding heterogeneous graph as a bridge, guiding the model's attention to the abnormal regions of the image, thus enhancing abnormal visual features. SIFF retrieves similar instances and integrates their abnormal visual features, thereby augmenting the representation of abnormal regions while acquiring a more comprehensive under-standing of the abnormal information. Experiments conducted on the IU X-ray and MIMIC-CXR medical imaging datasets demonstrate that the proposed method performs well on the BLEU evaluation metrics, achieving BLEU-1 to BLEU-4 scores of 0.539, 0.353, 0.265, and 0.193 respectively on the IU X-ray dataset. Additionally, it excels in METEOR and ROUGE-L metrics, indicating that the proposed method outperforms existing methods in terms of NLG metrics as well as the accuracy and completeness of the generated reports.

摘要： 医学影像报告自动生成任务存在影像对比度低、异常区域较小的难题，仅依靠影像信息难以精准刻画异常特征，因此如何引入外部知识来增强视觉表征成为解决问题的关键。此外，异常特征的共现关系复杂，依赖单一样本难以捕捉，如何利用相似实例建模共现模式至关重要。针对上述挑战，本文提出一种相似实例引导下融合异质图的医学影像报告生成方法，包括结合异质图的图像特征记忆模块和相似实例特征融合模块。结合异质图的图像特征记忆模块提取报告实体关系，构建报告对应异质图为桥梁，引导模型关注图像异常区域，增强异常视觉特征；相似实例特征融合模块检索相似实例，融合相似实例的异常视觉特征，增强异常区域特征表达的同时，获取更全面的异常信息。在 IU X-ray 和 MIMIC-CXR 这两个医学影像数据集上进行的实验评估显示，所提方法在 BLEU 系列评分指标上表现优秀，IU X-ray上B1~B4分别为0.539，0.353，0.265，0.193。同时，该方法在 METEOR 和 ROUGE-L 指标上的表现同样卓越。实验结果表明，所提方法在自然语言生成指标和生成报告的准确性、完整性方面优于现有主流方法，证明了方法的有效性。