基于多提示和图文对比学习的服装检索

doi:10.19678/j.issn.1000-3428.0069773

摘要/Abstract

摘要：

随着多模态学习的不断发展, 图像检索领域也面临新的机遇和挑战。现有的服装检索模型大多基于卷积神经网络或者Transformer的单模态模型实现, 忽略了图像对应的丰富文本信息, 模型能学习到的特征相对单一。为此, 提出一种基于多提示和图文对比学习的服装检索方法。引入图像文本多提示学习, 引导多模态大模型FashionCLIP学习服装的多维高语义多模态特征, 为提高模型的检索能力以及充分挖掘多模态模型的检索潜力, 分两阶段优化模型。第一阶段冻结图像和文本编码器, 通过图像文本交叉熵损失函数优化文本提示; 第二阶段冻结文本提示和文本编码器, 通过三元组损失、分类损失和图像文本交叉熵损失函数优化图像提示和图像编码器。在淘宝直播多模态视频商品检索数据集WAB上的域内检索和跨域检索实验结果表明: 该方法在域内检索的均值平均精度(mAP)和Rank-1相对于传统方法至少提升6.1和3.5百分点, 在跨域检索的mAP和Rank-1相对于传统方法至少提升8.4和6.4百分点, 检索性能得到了显著提升, 证明了图文对比学习在服装检索领域的潜力。

关键词: 服装检索, 图文对比学习, 预训练模型, 跨模态检索, 提示学习

Abstract:

With the continuous development of multimodal learning, the field of image retrieval is facing new opportunities and challenges. Most existing clothing retrieval models are based on convolutional neural networks or a Transformer's unimodal retrieval, ignoring the rich textual information corresponding to images. Moreover, the features that the model can learn tend to be relatively single. This study proposes a clothing retrieval method based on multiple prompts and contrastive image-text learning. This study introduces image and text multiprompt learning to guide a multimodal large model, called FashionCLIP, in learning the multidimensional, high semantic, and multimodal features of clothing. To improve the retrieval ability of the model and fully mine its multimodal potential, the model is optimized in two stages. In the first stage, the image and text encoders are frozen and the text prompt is optimized using image and text cross-entropy loss functions. In the second stage, the text prompt and text encoder are frozen, and the image prompt and image encoder are optimized using triple loss, classification loss, and image and text cross-entropy loss functions. Both intra- and cross-domain retrieval experiments were conducted on the Taobao Live multimodal video product retrieval dataset, known as WAB. The experimental results show that the mean Average Precision (mAP) of this method for intra-domain retrieval is improved by at least 6.1 percentage points compared to traditional models, and Rank-1 is improved by at least 3.5 percentage points compared to traditional models. This method improves the mAP compared to traditional models by at least 8.4 percentage points and Rank-1 by at least 6.4 percentage points in cross-domain retrieval. Additionally, the retrieval results are significantly improved, demonstrating the potential for contrastive learning in the field of clothing retrieval.

Key words: clothing retrieval, contrastive image-text learning, pre-trained model, cross-modal retrieval, prompt learning

孙圆, 王康平, 赵鸣博. 基于多提示和图文对比学习的服装检索[J]. 计算机工程, 2026, 52(2): 322-330.

SUN Yuan, WANG Kangping, ZHAO Mingbo. Clothing Retrieval Based on Multiple Prompts and Contrastive Image-Text Learning[J]. Computer Engineering, 2026, 52(2): 322-330.

https://www.ecice06.com/CN/Y2026/V52/I2/322

图/表 11

图1 多提示图文对比学习模型和图文对比学习模型结构

Fig.1 Structures of multi-prompts contrastive text-image learning model and contrastive text-image learning model

图2 数据集类别及标注示意图

Fig.2 Schematic diagram of dataset categories and annotations

图3 基线模型和本文模型在域内检索和跨域检索的对比结果

Fig.3 Comparison results between the baseline model and the proposed model for intra-domain and cross-domain retrieval

图4 可视化注意力图

Fig.4 Visualization of attention maps

图5 超参数λ_img对模型性能的影响

Fig.5 Impact of hyperparameter λ_img on model performance

图6 超参数λ_id对模型性能的影响

Fig.6 Impact of hyperparameter λ_id on model performance

图7 不同属性服装的模型可视化结果

Fig.7 Visualization results of models for clothing with different attributes

参考文献 26

1	SHARMA V, MURRAY N, LARLUS D, et al. Unsupervised meta-domain adaptation for fashion retrieval[C]//Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV). Washington D.C., USA: IEEE Press, 2021: 1-10.
2	KIAPOUR M H, HAN X, LAZEBNIK S, et al. Where to buy it: matching street clothing photos in online shops[C]//Proceedings of the IEEE International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2015: 3343-3351.
3	ZHU S M , ZOU X X , QIAN J J , et al. Learning structured relation embeddings for fine-grained fashion attribute recognition. IEEE Transactions on Multimedia, 2023, 26, 1652- 1664.
4	GU J Y, WANG K, LUO H, et al. MSINet: twins contrastive search of multi-scale interaction for object ReID[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 19243-19253.
5	姜爱萍, 刘骊, 付晓东, 等. 跨模态时尚检索的服装分层特征表示和关联学习. 计算机辅助设计与图形学学报, 2025, 37 (4): 654- 667.
	JIANG A P , LIU L , FU X D , et al. Clothing hierarchical feature representation and association learning for cross-modal fashion retrieval. Journal of Computer-Aided Design & Computer Graphics, 2025, 37 (4): 654- 667.
6	LIU Z W, LUO P, QIU S, et al. DeepFashion: powering robust clothes recognition and retrieval with rich annotations[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2016: 1096-1104.
7	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2024-03-11]. https://arxiv.org/pdf/1409.1556v1.
8	WANG Z H, GU Y J, ZHANG Y, et al. Clothing retrieval with visual attention model[C]//Proceedings of the IEEE Visual Communications and Image Processing (VCIP). Washington D.C., USA: IEEE Press, 2017: 1-4.
9	韩华, 黄丽, 田瑾, 等. 基于双中间模态的四流网络跨模态行人重识别. 计算机工程, 2023, 49 (8): 302- 309. doi: 10.19678/j.issn.1000-3428.0065333
	HAN H , HUANG L , TIAN J , et al. Cross-modality person re-identification using four-stream network based on dual-intermediate modalities. Computer Engineering, 2023, 49 (8): 302- 309. doi: 10.19678/j.issn.1000-3428.0065333
10	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 6000-6010.
11	WANG X X , JIANG B , WANG X , et al. Rethinking batch sample relationships for data representation: a batch-graph Transformer based approach. IEEE Transactions on Multimedia, 2023, 26, 1578- 1588.
12	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[EB/OL]. [2024-03-11]. https://www.semanticscholar.org/paper/An-Image-is-Worth-16x16-Words%3A-Transformers-for-at-Dosovitskiy-Beyer/268d347e8a55b5eb82fb5e7d2f800e33c75ab18a.
13	HE S T, LUO H, WANG P C, et al. TransReID: Transformer-based object re-identification[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2021: 14993-15002.
14	安国成, 江波, 王晓龙, 等. 基于拓展图文对比学习的多模态语义对齐. 计算机工程, 2024, 50 (11): 152- 162. doi: 10.19678/j.issn.1000-3428.0069468
	AN G C , JIANG B , WANG X L , et al. Multi-modal semantic alignment based on extended image-text contrastive learning. Computer Engineering, 2024, 50 (11): 152- 162. doi: 10.19678/j.issn.1000-3428.0069468
15	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[EB/OL]. [2024-03-11]. https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language.pdf.
16	CHIA P J, ATTANASIO G, BIANCHI F, et al. FashionCLIP: connecting language and images for product representations[EB/OL]. [2024-03-11]. https://arxiv.org/pdf/2204.03972v2.
17	ZHOU K Y , YANG J K , LOY C C , et al. Learning to prompt for vision-language models. International Journal of Computer Vision, 2022, 130 (9): 2337- 2348. doi: 10.1007/s11263-022-01653-1
18	ZHOU K Y, YANG J K, LOY C C, et al. Conditional prompt learning for vision-language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 16795-16804.
19	LI S Y, SUN L, LI Q L. CLIP-ReID: exploiting vision-language model for image re-identification without concrete text labels[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2023: 1405-1413.
20	Tianchi. Watch and buy 2021[EB/OL].[2024-03-11]. https://tianchi.aliyun.com/dataset/75730.
21	邓远飞, 李加伟, 蒋运承. 基于知识注入提示学习的专利短语相似度计算. 计算机工程, 2024, 50 (4): 294- 302. doi: 10.19678/j.issn.1000-3428.0067595
	DENG Y F , LI J W , JIANG Y C . Similarity computation of patent phrases based on knowledge injection prompt learning. Computer Engineering, 2024, 50 (4): 294- 302. doi: 10.19678/j.issn.1000-3428.0067595
22	周炫余, 吴莲华, 郑勤华, 等. 联合语义提示和记忆增强的弱监督跳绳视频异常检测方法. 计算机工程, 2024, 50 (7): 87- 95. doi: 10.19678/j.issn.1000-3428.0069415
	ZHOU X Y , WU L H , ZHENG Q H , et al. Weakly supervised video anomaly detection method for rope skipping combined with semantic prompts and memory enhancement. Computer Engineering, 2024, 50 (7): 87- 95. doi: 10.19678/j.issn.1000-3428.0069415
23	LESTER B, AL-RFOU R, CONSTANT N. The power of scale for parameter-efficient prompt tuning[EB/OL].[2024-03-11]. https://www.semanticscholar.org/paper/The-Power-of-Scale-for-Parameter-Efficient-Prompt-Lester-Al-Rfou/ffdbd7f0b03b85747b001b4734d5ee31b5229aa4.
24	HERMANS A, BEYER L, LEIBE B. In defense of the triplet loss for person re-identification[EB/OL].[2024-03-11]. https://arxiv.org/abs/1703.07737.
25	ZHUANG Z J , WEI L H , XIE L X , et al. Rethinking the distribution gap of person re-identification with camera-based batch normalization. Berlin, Germany: Springer International Publishing, 2020.
26	XIE W, LI X H, CAO C C, et al. ViT-CX: causal explanation of vision Transformers[EB/OL].[2024-03-11]. https://arxiv.org/abs/2211.03064.

[1]	杨钰雪, 何甜, 樊京杭, 刘瑞英, 李腾. 基于交叉注意力与特征聚合的跨模态图文检索研究[J]. 计算机工程, 2026, 52(2): 311-321.
[2]	曹渝昆, 王天浩, 李云峰, 陈明, 李晶晶, 刘元旻. 基于关系感知图神经网络的Text-to-SQL方法[J]. 计算机工程, 2025, 51(9): 129-138.
[3]	艾传鲜, 郭军军, 尹兆良. 基于层级软提示交互融合的少样本事件方面类别检测方法[J]. 计算机工程, 2025, 51(9): 120-128.
[4]	黄琦强, 安国成, 熊刚. 基于视觉-语言预训练模型的开集交通目标检测算法[J]. 计算机工程, 2025, 51(6): 375-384.
[5]	郭俊辰, 马御棠, 相艳, 赵学东, 郭军军. 基于Prompt打分的实体链接方法[J]. 计算机工程, 2025, 51(3): 334-341.
[6]	王庆丰, 李旭, 姚春龙, 程腾腾. 面向研究生招生咨询的中文Text-to-SQL模型[J]. 计算机工程, 2025, 51(3): 362-368.
[7]	朱红, 王阔然, 朱彤. 基于多侧面信息表征联合的实体相似性度量及对齐方法[J]. 计算机工程, 2025, 51(3): 64-75.
[8]	饶东宁, 许正辉, 梁瑞仕. 基于知识库问答的回答生成研究[J]. 计算机工程, 2025, 51(2): 94-101.
[9]	姚利峰, 蔡满春, 朱懿, 陈咏豪, 张溢文. 基于字节编码与预训练任务的加密流量分类模型[J]. 计算机工程, 2025, 51(2): 188-201.
[10]	刘海, 石佛波, 张昭理, 何嘉文, 李家豪. 基于文本和多视角局部结构特征的知识图谱推理[J]. 计算机工程, 2025, 51(11): 80-89.
[11]	路悦, 周翔宇, 张世周, 梁国强, 邢颖慧, 程德, 张艳宁. 基于预训练的持续学习方法综述(特邀)[J]. 计算机工程, 2025, 51(10): 1-17.
[12]	王皓嘉, 邓勇舰, 刘婷婷, 杨震. 基于DINO先验的混合特征面部表情识别模型[J]. 计算机工程, 2025, 51(10): 284-294.
[13]	费涛, 艾山·吾买尔, 杜文旭, 朱翠翠. 基于Squeezeformer的多颗粒度多方面发音质量评测方法[J]. 计算机工程, 2025, 51(1): 81-87.
[14]	魏嵬, 丁香香, 郭梦星, 杨钊, 刘辉. 文本相似度计算方法综述[J]. 计算机工程, 2024, 50(9): 18-32.
[15]	杨冬菊, 黄俊涛. 基于大语言模型的中文科技文献标注方法[J]. 计算机工程, 2024, 50(9): 113-120.

选择文件类型/文献管理软件名称

选择包含的内容