Research on Cross-Modal Image-Text Retrieval Based on Cross Attention and Feature Aggregation

doi:10.19678/j.issn.1000-3428.0070119

Abstract

Abstract: Image-text retrieval has become an important research direction in cross modal fields. However, the existing methods of aggregating multiple modal features face two major challenges: insufficient feature alignment between modalities and semantic representation loss within modalities. A cross modal image-text retrieval model based on cross attention and feature aggregation is proposed to address the problem of representation of feature information within modalities. This model includes modules such as image and text feature extraction, cross attention, feature pooling, and feature fusion. It combines the triplet loss function to mine local information in images and text, for obtaining image and text feature representations with deep semantic relationships. The model adopts an attention fusion strategy, which regulates the fusion of fine-grained features between images and texts using learnable weight parameters. A feature pooling module that aggregates image region features and text sequence features separately, learns weight parameters through neural networks, and combines multiple similarities to guide model learning is designed. This module can flexibly handle the features of variable length sequences of images and text, enhancing the ability of the model to capture cross modal information. Comparative experiments conducted on the public datasets MS COCO and Flickr 30k, reveal that compared with various image and text retrieval models, this model has higher retrieval performance. It has advantages in semantic feature pooling and dimensionality reduction, providing new concepts for cross modal feature fusion.

Key words: cross-modal retrieval, cross attention, image-text matching, feature pooling, feature fusion

摘要： 目前,图文检索已经成为跨模态领域的一个重要研究方向,但现有的将多种模态特征聚合的方式面临着模态间特征对齐不充分和模态内语义表征损失的两大挑战。针对跨模态检索领域模态内特征信息的表征问题,提出一种基于交叉注意力与特征聚合的跨模态图文检索模型。该模型包含图文特征提取、交叉注意力、特征池化、特征融合等模块,结合三元组损失函数挖掘图文局部信息,以获得具有深层次语义关系的图文特征表示。模型采用注意力融合策略,通过可学习权重参数调控图像与文本细粒度特征的融合。设计一种特征池化模块,分别聚合图像区域特征和文本序列特征,并通过神经网络学习权重参数,结合多重相似度共同指导模型学习,该模块可以灵活地处理图文变长序列的特征,增强模型对跨模态信息的捕捉能力。在公共数据集MS COCO和Flickr 30k上进行对比实验,结果表明,与多种图文检索模型相比,该模型在同类方法中检索性能更高,其在语义特征池化降维方面具有优势,为跨模态特征融合提供了新思路。

关键词: 跨模态检索, 交叉注意力, 图文匹配, 特征池化, 特征融合

CLC Number:

TP391

YANG Yuxue, HE Tian, FAN Jinghang, LIU Ruiying, LI Teng. Research on Cross-Modal Image-Text Retrieval Based on Cross Attention and Feature Aggregation[J]. Computer Engineering, 2026, 52(2): 311-321.

杨钰雪, 何甜, 樊京杭, 刘瑞英, 李腾. 基于交叉注意力与特征聚合的跨模态图文检索研究[J]. 计算机工程, 2026, 52(2): 311-321.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0070119

https://www.ecice06.com/EN/Y2026/V52/I2/311

References

[1] LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[2] ZHANG Y, JIN R, ZHOU Z H. Understanding bag-of-words model: a statistical framework[J]. International Journal of Machine Learning and Cybernetics, 2010, 1(1): 43-52.
[3] JELODAR H, WANG Y L, YUAN C, et al. Latent Dirichlet Allocation (LDA) and topic modeling: models, applications, a survey[J]. Multimedia Tools and Applications, 2019, 78(11): 15169-15211.
[4] HARDOON D R, SZEDMAK S, SHAWE-TAYLOR J. Canonical correlation analysis: an overview with application to learning methods[J]. Neural Computation, 2004, 16(12): 2639-2664.
[5] ZHENG W M, ZHOU X Y, ZOU C R, et al. Facial expression recognition using Kernel Canonical Correlation Analysis (KCCA)[J]. IEEE Transactions on Neural Networks, 2006, 17(1): 233-238.
[6] BENTON A, KHAYRALLAH H, GUJRAL B, et al. Deep generalized canonical correlation analysis[C]//Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019).[S.l.]:ACL,2019: 1-6.
[7] 高迪辉, 盛立杰, 许小冬, 等. 图文跨模态检索的联合特征方法[J]. 西安电子科技大学学报, 2024, 51(4): 128-138. GAO D H, SHENG L J, XU X D, et al. Joint feature approach for image-text cross-modal retrieval[J]. Journal of Xidian University, 2024, 51(4): 128-138. (in Chinese)
[8] LU J S, BATRA D, PARIKH D, et al. ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[EB/OL].[2024-05-05]. https://arxiv.org/abs/1908.02265.
[9] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[EB/OL].[2024-05-05]. https://arxiv.org/abs/2103.00020.
[10] LI J N, LI D X, XIONG C M, et al. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation[EB/OL].[2024-05-05]. https://arxiv.org/abs/2201.12086.
[11] LI J N, LI D X, SAVARESE S, et al. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models[C]//Proceedings of the 40th International Conference on Machine Learning. New York,USA:ACM Press, 2023: 19730-19742.
[12] LEE K H, CHEN X, HUA G, et al. Stacked cross attention for image-text matching[EB/OL].[2024-05-05]. https://arxiv.org/abs/1803.08024.
[13] ZHANG K, MAO Z D, WANG Q, et al. Negative-aware attention framework for image-text matching[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C.,USA:IEEE Press,2022: 15640-15649.
[14] YANG J Y, DUAN J L, TRAN S, et al. Vision-language pre-training with triple contrastive learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C.,USA:IEEE Press,2022: 15650-15659.
[15] LIU Z, PEI X L, GAO S S, et al. Perceive, reason, and align: context-guided cross-modal correlation learning for image-text retrieval[J]. Applied Soft Computing, 2024, 154: 111395.
[16] KRISHNA R, ZHU Y K, GROTH O, et al. Visual genome: connecting language and vision using crowd sourced dense image annotations[J]. International Journal of Computer Vision, 2017, 123(1): 32-73.
[17] REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149.
[18] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL].[2024-05-05]. https://arxiv.org/abs/1810.04805.
[19] CHEN J C, HU H X, WU H, et al. Learning the best pooling strategy for visual semantic embedding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C.,USA:IEEE Press,2021: 15784-15793.
[20] DELIÈGE A, ISTASSE M, KUMAR A, et al. Ordinal pooling[C]//Proceedings of the 30th British Machine Vision Conference. Cardiff, UK: BMVA Press, 2019: 76.
[21] KARPATHY A, LI F F. Deep visual-semantic alignments for generating image descriptions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C.,USA:IEEE Press,2015: 3128-3137.
[22] LI K P, ZHANG Y L, LI K, et al. Visual semantic reasoning for image-text matching[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C.,USA:IEEE Press,2019: 4653-4661.
[23] MITHUN N C, PANDA R, PAPALEXAKIS E E, et al. Webly supervised joint embedding for cross-modal image-text retrieval[C]//Proceedings of the 26th ACM International Conference on Multimedia. New York,USA:ACM Press,2018: 1856-1864.
[24] JI Z, CHEN K X, HE Y Q, et al. Heterogeneous memory enhanced graph reasoning network for cross-modal retrieval[J]. Science China Information Sciences, 2022, 65(7): 172104.
[25] ZHENG Z D, ZHENG L, GARRETT M, et al. Dual-Path convolutional image-text embeddings with instance loss[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2020, 16(2): 1-23.
[26] HUANG Y, WU Q, SONG C F, et al. Learning semantic concepts and order for image and sentence matching[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C.,USA:IEEE Press,2018: 6163-6171.
[27] CHEN H, DING G G, LIU X D, et al. IMRAM: iterative matching with recurrent attention memory for cross-modal image-text retrieval[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C.,USA:IEEE Press,2020: 12652-12660.
[28] 杨晓宇, 李超, 陈舜尧, 等. 基于Transformer的图文跨模态检索算法[J]. 计算机科学, 2023, 50(4): 141-148. YANG X Y, LI C, CHEN S Y, et al. Text-image cross-modal retrieval based on Transformer[J]. Computer Science, 2023, 50(4): 141-148. (in Chinese)
[29] 梁彦鹏, 刘雪儿, 马忠贵, 等. 嵌入共识知识的因果图文检索方法[J]. 工程科学学报, 2024(2): 317-328. LIANG Y P, LIU X E, MA Z G, et al. Causal image-text retrieval embedded with consensus knowledge[J]. Chinese Journal of Engineering, 2024(2): 317-328. (in Chinese)
[30] 廖律超, 邹伟东, 杨佳龙, 等. 基于注意力机制和微分跟踪器的宽度学习系统[J]. 深圳大学学报(理工版), 2024, 41(5): 583-593. LIAO L C, ZOU W D, YANG J L, et al. Broad learning system based on attention mechanism and tracking differentiator[J]. Journal of Shenzhen University (Science and Engineering), 2024, 41(5): 583-593. (in Chinese)

Please choose a citation manager

Content to export