基于双分支特征融合的医学报告生成方法

doi:10.19678/j.issn.1000-3428.0065065

摘要/Abstract

摘要： 医学图像的全局特征在基于深度学习的医学影像报告自动生成任务中发挥着重要作用，传统方法通常仅使用单分支卷积神经网络提取图像语义特征，注重局部细节特征的提取，但缺乏对医学图像全局特征的关注。提出一种新的医学影像报告生成方法DBFFN，基于双分支特征融合，结合卷积神经网络与视觉Transformer各自在图像特征提取上的优势，分别提取给定医学图像的全局特征和局部特征，在计算图像全局关系的同时关注局部细微的语义信息。针对医学图像的特征融合问题，设计一种多尺度特征融合模块，对来自两个分支的图像特征进行自适应尺度对齐，并结合矩阵运算方法和空间信息增强方法有效融合全局特征与局部特征内部包含的语义信息。在IU-X-Ray数据集上的实验结果表明，DBFFN方法的BLEU-1～BLEU-4，METEOR，ROUGE-L指标平均值分别为0.496，0.331，0.234，0.170，0.214，0.370，优于HRNN、HRGR、CMAS-RL等方法，在医学影像报告自动生成任务上具有有效性。

关键词: 医学影像报告生成, 全局特征, 局部特征, 特征融合, 图像-文本生成

Abstract: The global features of medical images play an important role in automatic medical report generation tasks based on deep learning models. However，in previous methods，only a single-branch convolutional neural network is adopted to extract image semantic features.These methods focus on local detailed features，ignoring the global features of medical images. Therefore，a new medical imaging report generation method called DBFFN，which is based on dual-branch feature fusion，is proposed to extract the global and local features of given medical images，thereby combining the advantages of Convolutional Neural Network（CNN） and visual Transformer in image feature extraction.The local subtle semantic information is considered when calculating the global relationship of the image，whereby two components are merged to complement each other.In addition，a Multi-Scale Feature Fusion Module（MSFFM） is designed for the feature fusion of medical images. First，the extracted features from the two branches are subjected to adaptive scale alignment. Subsequently，matrix operational and spatial information enhancement methods are combined to effectively fuse the semantic information contained in the global and local features. Through experiments on the IU-X-Ray data set，the average values of the optimal scores of DBFFN using the BLEU-1 to BLEU-4，METEOR，and ROUGE-L metrics reach 0.496，0.331，0.234，0.170，0.214，and 0.370，respectively，which is superior to the results obtained using methods such as HRNN，HRGR，and CMAS-RL，proving the effectiveness of DBFFN during the automatic medical reports generation task.

Key words: medical imaging report generation, global feature, local feature, feature fusion, image-text generation

中图分类号:

TP391.41

沈秀轩, 吴春雷, 冯叶棋, 程铭, 张俊三, 朱杰. 基于双分支特征融合的医学报告生成方法[J]. 计算机工程, 2023, 49(6): 274-283,291.

SHEN Xiuxuan, WU Chunlei, FENG Yeqi, CHENG Ming, ZHANG Junsan, ZHU Jie. Medical Report Generation Method Based on Dual-Branch Feature Fusion[J]. Computer Engineering, 2023, 49(6): 274-283,291.

http://www.ecice06.com/CN/Y2023/V49/I6/274

图/表 20

20230615170832

20230615170836

20230615170839

20230615170842

20230615170846

20230615170849

20230615170852

20230615170855

20230615170859

20230615170902

20230615170906

20230615170909

20230615170913

20230615170916

20230615170919

20230615170922

20230615170926

20230615170930

20230615170933

20230615170937

参考文献

[1] JING B,XIE P,XING E.On the automatic generation of medical imaging reports[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.[S.l.]:Association for Computational Linguistics,2018:2577-2586.
[2] VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems.New York,USA:ACM Press,2017:6000-6010.
[3] DOSOVITSKIY A,BEYER L,KOLESNIKOV A,et al.An image is worth 16×16 words:transformers for image recognition at scale[EB/OL].[2022-5-20].https://www.xueshufan.com/publication/3119786062.
[4] LIU Z,WANG Y,HAN K,et al.Post-training quantization for vision transformer[EB/OL].[2022-05-20].https://arxiv.org/abs/2106.14156.
[5] YUAN Y,FU R,HUANG L,et al.HRFormer:high-resolution vision transformer for dense predict[EB/OL].[2022-5-20].https://openreview.net/forum?id=DF8LCjR03tX.
[6] SNOEK C G M,WORRING M,SMEULDERS A W M.Early versus late fusion in semantic video analysis[C]//Proceedings of the 13th Annual ACM International Conference on Multimedia.New York,USA:ACM Press,2005:399-402.
[7] CHAIB S,LIU H,GU Y F,et al.Deep feature fusion for VHR remote sensing scene classification[J].IEEE Transactions on Geoscience and Remote Sensing,2017,55(8):4775-4784.
[8] DONG H,PAN J S,XIANG L,et al.Multi-scale boosted dehazing network with dense feature fusion[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2020:2154-2164.
[9] LIU C J,WECHSLER H.A shape and texture based enhanced fisher classifier for face recognition[J].IEEE Transactions on Image Processing,2001,10(4):598-608.
[10] YANG J,YANG J.Generalized K-L transform based combined feature extraction[J].Pattern Recognition,2002,35(1):295-297.
[11] YANG J,YANG J,ZHANG D,et al.Feature fusion:parallel strategy vs.serial strategy[J].Pattern Recognition,2003,36(6):1369-1381.
[12] LIU X W,ZHU X Z,LI M M,et al.Late fusion incomplete multi-view clustering[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,41(10):2410-2423.
[13] TRONG V H,GWANG-HYUN Y,VU D T,et al.Late fusion of multimodal deep neural networks for weeds classification[J].Computers and Electronics in Agriculture,2020,175(6):56-67.
[14] 李娜,顾庆,姜枫,等.一种基于卷积神经网络的砂岩显微图像特征表示方法[J].软件学报,2020,31(11):3621-3639.LI N,GU Q,JIANG F,et al.Feature representation method of microscopic sandstone images based on convolutional neural network[J].Journal of Software,2020,31(11):3621-3639.(in Chinese)
[15] 袁单飞,陈慈发,董方敏.基于多尺度分割的图像识别残差网络研究[J].计算机工程,2022,48(5):258-262,271.YUAN D F,CHEN C F,DONG F M.Research on residual network of image recognition based on multiscale split[J].Computer Engineering,2022,48(5):258-262,271.(in Chinese)
[16] MATSOUKAS C,HASLUM J F,SÖDERBERG M,et al.Is it time to replace CNNs with transformers for medical images?[EB/OL].[2022-5-20].https://arxiv.org/abs/2108.09038.
[17] VINYALS O,TOSHEV A,BENGIO S,et al.Show and tell:a neural image caption generator[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2015:3156-3164.
[18] NAJDENKOSKA I,ZHEN X T,WORRING M,et al.Variational topic inference for chest X-ray report generation[M]//BRUIJNE M D,CATTIN P C,COTIN S,et al.Medical image computing and computer assisted intervention.Berlin,Germany:Springer International Publishing,2021:625-635.
[19] HAN Z,WEI B,XI X,et al.Unifying neural learning and symbolic reasoning for spinal medical report generation[J].Medical Image Analysis,2021,67:101872.
[20] YANG X,GIREESH N,XING E,et al.XRayGAN:consistency-preserving generation of X-ray images from radiology reports[EB/OL].[2022-05-20].https://arxiv.org/abs/2006.10552.
[21] HUANG J H,YANG C H H,LIU F Y,et al.DeepOpht:medical report generation for retinal images via deep models and visual explanation[C]//Proceedings of IEEE Winter Conference on Applications of Computer Vision.Washington D.C.,USA:IEEE Press,2021:2441-2451.
[22] KRAUSE J,JOHNSON J,KRISHNA R,et al.A hierarchical approach for generating descriptive image paragraphs[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA.IEEE Press,2017:3337-3345.
[23] LI C Y,LIANG X,HU Z,et al.Hybrid retrieval-generation reinforced agent for medical image report generation[EB/OL].[2022-05-20].https://arxiv.org/abs/1805.08298.
[24] JING B Y,WANG Z Y,XING E.Show,describe and conclude:on exploiting the structure information of chest X-ray reports[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Stroudsburg,USA:Association for Computational Linguistics,2019:6570-6580.
[25] ZHANG Y X,WANG X S,XU Z Y,et al. When radiology report generation meets knowledge graph[EB/OL].[2022-05-20]https://arxiv.org/abs/2002.08277v1.
[26] XU W,QI C,XU Z,et al.Reinforced medical report generation with X-linear attention and repetition penalty[EB/OL].[2022-05-20].https://arxiv.org/abs/2011.07680.
[27] CHEN Z H,SONG Y,CHANG T H,et al.Generating radiology reports via memory-driven transformer[C]//Proceedings of 2020 Conference on Empirical Methods in Natural Language Processing.Stroudsburg,USA:Association for Computational Linguistics,2020:1439-1449.
[28] LIU F L,WU X,GE S,et al.Exploring and distilling posterior and prior knowledge for radiology report generation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2021:13748-13757.
[29] LIU F,YIN C,WU X,et al.Contrastive attention for automatic chest X-ray report generation[EB/OL].[2022-05-20].https://arxiv.org/abs/2106.06965.

选择文件类型/文献管理软件名称

选择包含的内容