摘要： 现有基于深度学习的多模态医学图像融合方法存在全局特征表示能力不足的问题。对此，提出一种基于局部全局特征耦合与交叉尺度注意的医学图像融合方法。该方法由编码器、融合规则和解码器三部分组成。编码器中采用并行的卷积神经网络（CNN）和Transformer双分支网络分别提取图像的局部特征与全局表示。在不同尺度下，通过特征耦合模块将CNN分支的局部特征嵌入Transformer分支的全局特征表示中，最大程度地结合互补特征，同时引入交叉尺度注意模块实现对多尺度特征表示的有效利用。编码器提取待融合原始图像的局部、全局以及多尺度特征表示，根据融合规则融合不同源图像的特征表示后再输入到解码器中生成融合图像。实验结果表明，与CBF、PAPCNN、IFCNN、DenseFuse和U2Fusion方法相比，该方法在特征互信息、空间频率、边缘信息传递因子、结构相似度、感知图像融合质量这5个评价指标上分别平均提高6.29%、3.58%、29.01%、5.34%、5.77%，融合图像保留了更清晰的纹理细节和更高的对比度，便于疾病的诊断与治疗。

关键词: 医学图像融合, 编码器-解码器网络, Transformer网络, 特征耦合, 交叉尺度注意

Abstract: To address the insufficient global feature representation in existing deep learning-based multimodal medical image fusion methods, this study proposes a medical image fusion method based on local-global feature coupling and cross-scale attention.The method comprises three parts:encoder, fusion rule, and decoder.In the encoder, parallel Convolutional Neural Network(CNN) and Transformer dual-branch networks are used to extract the local features and global representation of the image, respectively.At different scales, the local features of the CNN branch are embedded into the global feature representation of the Transformer branch through the feature coupling module for combining complementary features;simultaneously, a cross-scale attention module is introduced to effectively utilize multiscale feature representation.The encoder extracts the local, global, and multiscale feature representations of the original images to be fused, fuses the feature representations of different source images through fusion rules, and then inputs them into the decoder to generate the fused image.Experiments show that compared with CBF, PAPCNN, IFCNN, DenseFuse, and U2Fusion methods, the proposed method objectively improves the five evaluation indicators of feature mutual information, spatial frequency, edge information transfer factor, structural similarity, and perceptual image fusion quality by 6.29%, 3.58%, 29.01%, 5.34%, and 5.77%, respectively;subjectively, the fusion images obtained using this method retain clearer texture details and higher contrast, which is convenient for disease diagnosis and treatment.

Key words: medical image fusion, encoder-decoder network, Transformer network, feature coupling, cross-scale attention

中图分类号: 

• TP391