摘要：

现有基于Transformer架构的图像描述生成模型取得了较好的泛化性能, 然而, 大多数方法通常使用区域视觉特征进行编解码, 导致无法全面利用整幅图像的细粒度信息, 且存在视觉特征混淆问题。为此, 将全景分割引入图像描述生成过程, 使用基于全景分割的掩膜视觉特征代替区域视觉特征, 提出一种全景分割与多视觉特征协同的图像描述生成方法。该方法不仅可以有效解耦视觉表征, 而且能够充分结合掩膜视觉特征和网格视觉特征的优势, 提升图像描述生成的可解释性和描述性能。在MSCOCO标准数据集上进行定量和定性实验, 结果表明, 所提方法不仅可以显著提升现有模型的性能, 同时能够增强图像描述生成过程的可解释性, CIDEr和BLEU-4指标分别达到138.5和41。

关键词: 图像理解, 图像描述生成, 全景分割, 特征融合, 视觉编码

Abstract:

Due to their powerful sequence modeling capabilities, Transformer-based image captioning models have demonstrated remarkable performance. However, most of these models typically utilize region visual features to perform encoding and decoding, which cannot fully use the fine-grained information of the whole image, and this leads to visual feature confusion. Accordingly, we introduce panoptic segmentation into the Transformer-based image captioning model by replacing the region visual feature with mask visual features and propose a novel image captioning model based on multi-visual-feature fusion. Our model not only disentangles the region visual features effectively but also makes use of both mask and grid visual features to improve image captioning performance. We perform quantitative and qualitative experiments on the MSCOCO dataset, which demonstrate that our method significantly outperforms existing Transformer-based image captioning models. In addition, our model enhances the interpretability of the caption generation process, and more specifically, achieves CIDEr and BLEU-4 scores of 138.5 and 41, respectively.

Key words: image understanding, image description generation, panoptic segmentation, feature fusion, visual encoding