基于Transformer视觉特征融合的图像描述方法

doi:10.19678/j.issn.1000-3428.0068402

摘要/Abstract

摘要：

现有图像描述方法只利用区域型视觉特征生成描述语句, 忽略了网格型视觉特征的重要性, 并且均为两阶段方法, 从而影响了图像描述的质量。针对该问题, 提出一种基于Transformer视觉特征融合的端到端图像描述方法。首先, 在特征提取阶段, 利用视觉特征提取器提取出区域型视觉特征和网格型视觉特征; 其次, 在特征融合阶段, 通过视觉特征融合模块对区域型视觉特征和网格型视觉特征进行拼接; 最后, 将所有的视觉特征送入语言生成器中以生成图像描述。该方法各部分均基于Transformer模型实现, 实现了一阶段方法。在MS-COCO数据集上的实验结果表明, 所提方法能够充分利用区域型视觉特征与网格型视觉特征的优势, BLEU-1、BLEU-4、METEOR、ROUGE-L、CIDEr、SPICE指标分别达到83.1%、41.5%、30.2%、60.1%、140.3%、23.9%, 优于目前主流的图像描述方法, 能够生成更加准确和丰富的描述语句。

关键词: 图像描述, 区域型视觉特征, 网格型视觉特征, Transformer模型, 端到端训练

Abstract:

Existing image captioning methods only use regional visual features to generate description statements and ignore the importance of grid visual features. Moreover, as these methods are two-stage approaches, image captioning quality is affected. To address this issue, this study proposes an end-to-end image captioning method based on the visual feature fusion of Transformer. First, in the feature extraction stage, the visual feature extractor is used to extract regional and grid visual features. Second, in the feature fusion stage, the regional and grid visual features are concatenated using a visual feature fusion module. Finally, the visual features are sent to the language generator to realize image captioning. All components of the method are implemented based on the Transformer model, which is a one-stage method. The experimental results on the MS-COCO dataset show that the proposed method can fully utilize the respective advantages of regional and grid visual features, with the BLEU-1, BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE metrics reaching 83.1%, 41.5%, 30.2%, 60.1%, 140.3%, and 23.9%, respectively, indicating that the proposed method is superior to mainstream image captioning methods and can generate more accurate and rich description statements.

Key words: image captioning, regional visual features, grid visual features, Transformer model, end-to-end training

白雪冰, 车进, 吴金蔓, 陈玉敏. 基于Transformer视觉特征融合的图像描述方法[J]. 计算机工程, 2024, 50(8): 229-238.

Xuebing BAI, Jin CHE, Jinman WU, Yumin CHEN. Image Captioning Method Based on Transformer Visual Features Fusion[J]. Computer Engineering, 2024, 50(8): 229-238.

https://www.ecice06.com/CN/Y2024/V50/I8/229

图/表 12

图1 模型的整体框架

Fig.1 Overall framework of the model

图2 骨干网络结构

Fig.2 Backbone network structure

图3 提取区域型视觉特征的网络架构

Fig.3 Network architecture for extracting regional visual features

图4 提取网格型视觉特征的网络架构

Fig.4 Network architecture for extracting grid based visual features

图5 语言生成器架构

Fig.5 Language generator architecture

图6 调整后的多头注意力模块

Fig.6 Adjusted multi-head attention module

图7 参数量、计算量、推理时间的比较结果

Fig.7 Comparison results of parameter quantity, computational complexity, and inference time

图8 本文模型的图像描述与人工标注结果对比

Fig.8 Comparison between the image description of the model in this paper and the results of manual annotation

参考文献 36

1	LI Z X, LIN L, ZHANG C L, et al. A semi-supervised learning approach based on adaptive weighted fusion for automatic image annotation. ACM Transactions on Multimedia Computing, Communications, and Applications, 2021, 17(1): 1- 23.
2	衡红军, 范昱辰, 王家亮. 基于Transformer的多方面特征编码图像描述生成算法. 计算机工程, 2023, 49(2): 199- 205. URL
	HENG H J, FAN Y C, WANG J L. Multifaceted feature coding image caption generation algorithm based on Transformer. Computer Engineering, 2023, 49(2): 199- 205. URL
3	卓亚琦, 魏家辉, 李志欣. 基于双注意模型的图像描述生成方法研究. 电子学报, 2022, 50(5): 1123- 1130. URL
	ZHUO Y Q, WEI J H, LI Z X. Research on image captioning based on double attention model. Acta Electronica Sinica, 2022, 50(5): 1123- 1130. URL
4	CHEN F L, ZHANG D Z, HAN M L, et al. VLP: a survey on vision-language pre-training. Machine Intelligence Research, 2023, 20(1): 38- 56. doi: 10.1007/s11633-022-1369-5
5	YANG X, ZHANG H W, GAO C Y, et al. Learning to collocate visual-linguistic neural modules for image captioning. International Journal of Computer Vision, 2023, 131(1): 82- 100. doi: 10.1007/s11263-022-01692-8
6	FENG Y M, LAN L, ZHANG X, et al. AttResNet: attention-based ResNet for image captioning[C]// Proceedings of 2018 International Conference on Algorithms, Computing and Artificial Intelligence. New York, USA: ACM Press, 2018: 1-6.
7	HOSSAIN M Z, SOHEL F, SHIRATUDDIN M F, et al. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys, 2019, 51(6): 1- 36.
8	WANG D F, HU H F, CHEN D H. Transformer with sparse self-attention mechanism for image captioning. Electronics Letters, 2020, 56(15): 764- 766. doi: 10.1049/el.2020.0635
9	HUANG L, WANG W M, CHEN J, et al. Attention on attention for image captioning[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 4634-4643.
10	GUO L T, LIU J, ZHU X X, et al. Normalized and geometry-aware self-attention network for image captioning[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 10327-10336.
11	赵敬伟, 林珊玲, 梅婷, 等. 基于YOLACT与Transformer相结合的实例分割算法研究. 半导体光电, 2023, 44(1): 134- 140. URL
	ZHAO J W, LIN S L, MEI T, et al. Research on instance segmentation algorithm based on YOLACT and Transformer. Semiconductor Optoelectronics, 2023, 44(1): 134- 140. URL
12	MAO J H, XU W, YANG Y, et al. Deep captioning with multimodal Recurrent Neural Networks (m-RNN)[EB/OL]. [2023-08-05]. https://arxiv.org/abs/1412.6632.
13	VINYALS O, TOSHEV A, BENGIO S, et al. Show and tell: a neural image caption generator[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 3156-3164.
14	ANDERSON P, HE X D, BUEHLER C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 6077-6086.
15	李志欣, 魏海洋, 黄飞成, 等. 结合视觉特征和场景语义的图像描述生成. 计算机学报, 2020, 43(9): 1624- 1640. URL
	LI Z X, WEI H Y, HUANG F C, et al. Combine visual features and scene semantics for image captioning. Chinese Journal of Computers, 2020, 43(9): 1624- 1640. URL
16	宋井宽, 曾鹏鹏, 顾嘉扬, 等. 基于视觉区域聚合与双向协作的端到端图像描述生成. 软件学报, 2023, 34(5): 2152- 2169. URL
	SONG J K, ZENG P P, GU J Y, et al. End-to-end image captioning via visual region aggregation and dual-level collaboration. Journal of Software, 2023, 34(5): 2152- 2169. URL
17	PAN Y W, YAO T, LI Y H, et al. X-linear attention networks for image captioning[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 10968-10977.
18	SONG Z L, ZHOU X F, DONG L H, et al. Direction relation Transformer for image captioning[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York, USA: ACM Press, 2021: 5056-5064.
19	JIANG H Z, MISRA I, ROHRBACH M, et al. In defense of grid features for visual question answering[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 10264-10273.
20	WU M R, ZHANG X Y, SUN X S, et al. DIFNet: boosting visual information flow for image captioning[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 18020-18029.
21	RENNIE S J, MARCHERET E, MROUEH Y, et al. Self-critical sequence training for image captioning[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 1179-1195.
22	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[EB/OL]. [2023-08-05]. https://arxiv.org/abs/1405.0312.
23	KARPATHY A, LI F F. Deep visual-semantic alignments for generating image descriptions[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 3128-3137.
24	PAPINENI K, ROUKOS S, WARD T, et al. BLEU: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. [S. l. ]: ACL, 2001: 311-318.
25	BANERJEE S, LAVIE A. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments[EB/OL]. [2023-08-05]. https://aclanthology.org/W05-0909/.
26	LIN C Y. ROUGE: a package for automatic evaluation of summaries[EB/OL]. [2023-08-05]. https://aclanthology.org/W04-1013/.
27	VEDANTAM R, ZITNICK C L, PARIKH D. CIDEr: consensus-based image description evaluation[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 4566-4575.
28	ANDERSON P, FERNANDO B, JOHNSON M, et al. SPICE: semantic propositional image caption evaluation[EB/OL]. [2023-08-05]. https://arxiv. org/abs/1607.08822.
29	NGUYEN V Q, SUGANUMA M, OKATANI T. GRIT: faster and better image captioning Transformer using dual visual features[EB/OL]. [2023-08-05]. https://arxiv. org/abs/2207.09666.
30	LU J S, XIONG C M, PARIKH D, et al. Knowing when to look: adaptive attention via a visual sentinel for image captioning[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 3242-3250.
31	刘茂福, 施琦, 聂礼强. 基于视觉关联与上下文双注意力的图像描述生成方法. 软件学报, 2022, 33(9): 3210- 3222. URL
	LIU M F, SHI Q, NIE L Q. Image captioning based on visual relevance and context dual attention. Journal of Software, 2022, 33(9): 3210- 3222. URL
32	YAO T, PAN Y W, LI Y H, et al. Exploring visual relationship for image captioning[EB/OL]. [2023-08-05]. https://arxiv. org/abs/1809.07041.
33	CORNIA M, STEFANINI M, BARALDI L, et al. Meshed-memory Transformer for image captioning[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 10575-10584.
34	JI J Y, LUO Y P, SUN X S, et al. Improving image captioning by leveraging intra- and inter-layer global representation in Transformer network. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(2): 1655- 1663. doi: 10.1609/aaai.v35i2.16258
35	ZHANG X Y, SUN X S, LUO Y P, et al. RSTNet: captioning with adaptive attention on visual and non-visual words[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 15460-15469.
36	LUO Y P, JI J Y, SUN X S, et al. Dual-level collaborative Transformer for image captioning. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(3): 2286- 2293. doi: 10.1609/aaai.v35i3.16328

[1]	朱凯, 李理, 张彤, 江晟, 别一鸣. 基于Transformer的多阶段运动模糊图像修复网络[J]. 计算机工程, 2024, 50(9): 276-285.
[2]	王言国, 吕鹏远, 兰金江, 刘明哲, 秦冠军, 张硕桦, 周宇. 基于对抗训练与Transformer的风力发电机故障分类方法[J]. 计算机工程, 2024, 50(9): 377-384.
[3]	赵宏, 王枭. 基于Swin-Transformer的黑色素瘤图像病灶分割研究[J]. 计算机工程, 2024, 50(8): 249-258.
[4]	张亚洲, 和玉, 戎璐, 王祥凯. 基于上下文知识增强型Transformer网络的抑郁检测[J]. 计算机工程, 2024, 50(8): 75-85.
[5]	贺姗, 蔺素珍, 王彦博, 李大威. 基于特征融合的多波段图像描述生成方法[J]. 计算机工程, 2024, 50(6): 236-244.
[6]	侯颖, 杨林, 胡鑫, 贺顺, 宋婉莹, 赵谦. 基于SwinT-YOLOX模型的自动扶梯行人安全检测算法[J]. 计算机工程, 2024, 50(3): 277-289.
[7]	刘帅威, 李智, 王国美, 张丽. 基于Transformer和GAN的对抗样本生成算法[J]. 计算机工程, 2024, 50(2): 180-187.
[8]	甘陈敏, 唐宏, 杨浩澜, 刘小洁, 刘杰. 融合卷积收缩门控的生成式文本摘要方法[J]. 计算机工程, 2024, 50(2): 98-104.
[9]	池亚平, 岳梓岩, 林雨衡. 基于Transformer的SM4算法工作模式识别[J]. 计算机工程, 2023, 49(9): 109-117.
[10]	衡红军, 范昱辰, 王家亮. 基于Transformer的多方面特征编码图像描述生成算法[J]. 计算机工程, 2023, 49(2): 199-205.
[11]	孙懿, 高见, 顾益军. 融合一维Inception结构与ViT的恶意加密流量检测[J]. 计算机工程, 2023, 49(1): 154-162.
[12]	王会勇, 卢超, 张晓明. 基于小样本学习和语义信息的图像描述模型[J]. 计算机工程, 2021, 47(8): 260-270.
[13]	王涛, 刘超辉, 郑青青, 黄嘉曦. 基于单向Transformer和孪生网络的多轮任务型对话技术[J]. 计算机工程, 2021, 47(7): 55-58,66.
[14]	李雅红, 周海英, 徐少伟. 基于对象关系网状转换器的图像描述模型[J]. 计算机工程, 2021, 47(5): 197-204.
[15]	曹渝昆, 魏健强, 孙涛, 徐越. 基于IndRNN与BN的深层图像描述模型[J]. 计算机工程, 2021, 47(10): 194-200.

选择文件类型/文献管理软件名称

选择包含的内容