摘要：

目前, 图文检索已经成为跨模态领域的一个重要研究方向, 但现有的将多种模态特征聚合的方式面临着模态间特征对齐不充分和模态内语义表征损失的两大挑战。针对跨模态检索领域模态内特征信息的表征问题, 提出一种基于交叉注意力与特征聚合的跨模态图文检索模型。该模型包含图文特征提取、交叉注意力、特征池化、特征融合等模块, 结合三元组损失函数挖掘图文局部信息, 以获得具有深层次语义关系的图文特征表示。模型采用注意力融合策略, 通过可学习权重参数调控图像与文本细粒度特征的融合。设计一种特征池化模块, 分别聚合图像区域特征和文本序列特征, 并通过神经网络学习权重参数, 结合多重相似度共同指导模型学习, 该模块可以灵活地处理图文变长序列的特征, 增强模型对跨模态信息的捕捉能力。在公共数据集MS COCO和Flickr 30k上进行对比实验, 结果表明, 与多种图文检索模型相比, 该模型在同类方法中检索性能更高, 其在语义特征池化降维方面具有优势, 为跨模态特征融合提供了新思路。

关键词: 跨模态检索, 交叉注意力, 图文匹配, 特征池化, 特征融合

Abstract:

Image-text retrieval has become an important research direction in cross modal fields. However, the existing methods of aggregating multiple modal features face two major challenges: insufficient feature alignment between modalities and semantic representation loss within modalities. A cross modal image-text retrieval model based on cross attention and feature aggregation is proposed to address the problem of representation of feature information within modalities. This model includes modules such as image and text feature extraction, cross attention, feature pooling, and feature fusion. It combines the triplet loss function to mine local information in images and text, for obtaining image and text feature representations with deep semantic relationships. The model adopts an attention fusion strategy, which regulates the fusion of fine-grained features between images and texts using learnable weight parameters. A feature pooling module that aggregates image region features and text sequence features separately, learns weight parameters through neural networks, and combines multiple similarities to guide model learning is designed. This module can flexibly handle the features of variable length sequences of images and text, enhancing the ability of the model to capture cross modal information. Comparative experiments conducted on the public datasets MS COCO and Flickr 30k, reveal that compared with various image and text retrieval models, this model has higher retrieval performance. It has advantages in semantic feature pooling and dimensionality reduction, providing new concepts for cross modal feature fusion.

Key words: cross-modal retrieval, cross attention, image-text matching, feature pooling, feature fusion