作者投稿和查稿 主编审稿 专家审稿 编委审稿 远程编辑

计算机工程 ›› 2021, Vol. 47 ›› Issue (5): 197-204. doi: 10.19678/j.issn.1000-3428.0059517

• 图形图像处理 • 上一篇    下一篇

基于对象关系网状转换器的图像描述模型

李雅红, 周海英, 徐少伟   

  1. 中北大学 大数据学院, 太原 030051
  • 收稿日期:2020-09-14 修回日期:2020-11-08 发布日期:2020-10-13
  • 作者简介:李雅红(1996-),女,硕士研究生,主研方向为图像处理、模式识别;周海英(通信作者),教授、博士;徐少伟,硕士研究生。
  • 基金资助:
    国家自然科学基金(61672473)。

Image Description Model Based on Object Relation Mesh Transformer

LI Yahong, ZHOU Haiying, XU Shaowei   

  1. School of Data Science and Technology, North University of China, Taiyuan 030051, China
  • Received:2020-09-14 Revised:2020-11-08 Published:2020-10-13

摘要: 针对图像描述生成模型缺乏空间关系信息且图像特征利用不充分的问题,结合对象关系网状转换器,提出一种改进的图像描述模型。利用Faster R-CNN提取图像的外观和边界框特征,并将提取的特征输入到改进的转换器中经过编解码生成图像描述。通过将对象外观和边界框特征合并为关系特征的方式对编码器自我注意力层的注意力权值进行改进,以强化目标间的关联性。将编码器和解码器的连接设计为网状结构,从而充分利用图像特征。实验结果表明,与基于单一注意力的Top-down基线模型相比,该模型的BLUE@1和CIDEr评价指标值分别提高了7.6和3.7个百分点,显著提升了描述语句的准确性。

关键词: 图像描述模型, 注意力机制, 编码器和解码器, 对象关系, 网状转换器

Abstract: The existing image description generation models generally lack spatial relation information and do not fully utilize image features.To address the problem,this paper proposes an image description model based on object relation mesh transformer.The model employs Faster R-CNN to detect the features of the appearance and boundary box of the image,and inputs the detected features into the improved converter.In the transformer,the features are encoded and decoded to generate the image description.At the same time,the attention weight of the adaptive attention layer of the encoder is improved by integrating the features of object appearance and boundary box into relational features,which enhances the correlation between targets.In order to make full use of image features,the connection between the encoder and the decoder is designed as a mesh structure.The experimental results show that the score of the proposed model on BLUE@1 and CIDEr is 7.6 and 3.7 percentage points higher than that of the Top-down baseline model based on single attention,which demonstrates that the model significantly increases the accuracy of descriptive sentences.

Key words: image description model, attention mechanism, encoder and decoder, object relation, mesh transformer

中图分类号: