基于拓展图文对比学习的多模态语义对齐

doi:10.19678/j.issn.1000-3428.0069468

摘要/Abstract

摘要：

基于对比语言-图像的预训练(CLIP)方法在大规模图文数据上使双流架构下的模型能够较好地学习到统一的高级语义表征, 但CLIP模式仅约束图像-文本模态间的粗粒度语义对齐, 在同一模态下的语义表征仍需改进。为了使网络学习到更好的潜在统一语义表征, 提出一种基于拓展图文对比学习的多模态语义对齐方法。首先通过微调预训练的CLIP模型, 针对指定数据集优化语义表征, 设计双向匹配策略构造图文样本匹配拓扑图, 然后利用拓扑图中关联度更高的图文样本将对比学习进行拓展, 在图像-文本模态下进行粗粒度语义对齐, 同时在相同模态中进行细粒度调整, 并引入可学习参数调整各模态下的对比损失权重。通过在多个数据集下的实验结果表明, 该方法在不影响多模态语义对齐的性能下能够改进相同模态下的语义表征, 在分类、检索等下游任务上具有更好或相当的性能。

关键词: 多模态学习, 语义表征, 对比学习, 图文匹配, 图像分类

Abstract:

The current pre-training model, known as Contrastive Language-Image Pre-training (CLIP), facilitates a dual-stream architecture that allows the model to learn unified high-level semantic representations on large-scale text-image data. However, the CLIP model only enforces coarse-grained semantic alignment between image-text modalities, and semantic representation within the same modality requires further improvement. To assist the network in learning unified semantic representations more effectively, this study proposes a multimodal semantic alignment method based on extended image-text contrastive learning. First, by fine-tuning the pre-trained CLIP model, the semantic representation is optimized for the specified dataset, and a bidirectional matching strategy is designed to construct a text-image sample-matching topology graph. Contrastive learning is then extended using image-text samples with higher relevance in the topology graph, facilitating coarse-grained semantic alignment between image-text modalities. Fine-grained adjustments are performed within the same modality and learnable parameters are introduced to adjust the contrastive loss weights under each modality. Experiments on multiple datasets demonstrate that this method improves semantic representation within the same modality without affecting the performance of multimodal semantic alignment, and facilitates better or comparable performance in classification, retrieval, and other downstream tasks.

Key words: multi-modal learning, semantic representation, contrastive learning, image-text matching, image classification

安国成, 江波, 王晓龙, 戴军. 基于拓展图文对比学习的多模态语义对齐[J]. 计算机工程, 2024, 50(11): 152-162.

AN Guocheng, JIANG Bo, WANG Xiaolong, DAI Jun. Multi-modal Semantic Alignment Based on Extended Image-Text Contrastive Learning[J]. Computer Engineering, 2024, 50(11): 152-162.

https://www.ecice06.com/CN/Y2024/V50/I11/152

图/表 19

图1 图文对比学习框架

Fig.1 Framework of image-text comparison learning

图2 图文匹配环挖掘流程

Fig.2 Procedure of image-text matching rings mining

图3 图文匹配环数据样例

Fig.3 Sample of image-text matching rings data

图4 拓展图文对比学习

Fig.4 Extended Image-Text contrastive learning

图5 图像-文本跨模态训练损失

Fig.5 Image-text cross-modal training loss

图6 文本模态训练损失

Fig.6 Text modal training loss

图7 图像模态训练损失

Fig.7 Image modal training loss

图8 图像-文本跨模态训练损失

Fig.8 Image-text cross modal training loss

图9 图像模态训练损失

Fig.9 Image modal training loss

参考文献 37

1	NGIAM J, KHOSLA A, KIM M, et al. Multimodal deep learning[C]//Proceedings of the 28th International Conference on Machine Meaning. Washington D. C., USA: IEEE Press, 2011: 689-696.
2	SRIVASTAVA N, SALAKHUTDINOV R. Multimodal learning with deep Boltzmann machines. Journal of Machine Learning Research, 2014, 15, 2949- 2980. doi: 10.1162/NECO_a_00311
3	XU K, BA J L, KIROS R, et al. Show, attend and tell: neural image caption generation with visual attention[C]//Proceedings of the 32nd International Conference on Machine Learning. Washington D. C., USA: IEEE Press, 2015: 2048-2057.
4	LIU Z, LI J G, SHEN Z Q, et al. Learning efficient convolutional networks through network slimming[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 2736-2744.
5	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of NIPS'17. Cambridge, USA: MIT Press, 2017: 30-39.
6	XU P, ZHU X T, CLIFTON D A. Multimodal learning with transformers: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45 (10): 12113- 12132. doi: 10.1109/TPAMI.2023.3275156
7	吴志强, 解庆, 李琳, 等. 基于多模态融合的图神经网络推荐算法. 计算机工程, 2024, 50 (1): 91- 100. doi: 10.19678/j.issn.1000-3428.0066929
	WU Z Q, XIE Q, LI L, et al. Graph neural network recommendation algorithm based on multimodal fusion. Computer Engineering, 2024, 50 (1): 91- 100. doi: 10.19678/j.issn.1000-3428.0066929
8	LI X J, YIN X, LI C Y, et al. Oscar: object-semantics aligned pre-training for vision-language tasks. Berlin, Germany: Springer, 2020.
9	CHEN Y C, LI L J, YU L C, et al. UNITER: universal image-TExt representation learning[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 104-120.
10	KIM W, SON B, KIM I. Vilt: vision-and-language transformer without convolution or region supervision[C]// Proceedings of International Conference on Machine Learning. Washington D. C., USA: IEEE Press, 2021: 5583-5594.
11	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[EB/OL]. [2024-02-01]. https://arxiv.org/pdf/2010.11929.pdf.
12	TSAI Y H H, BAI S J, LIANG P P, et al. Multimodal transformer for unaligned multimodal language sequences[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics[EB/OL]. [2024-02-01]. https://arxiv.org/abs/1906.00295.
13	LU J S, BATRA D, PARIKH D, et al. ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[EB/OL]. [2024-02-01]. https://arxiv.org/abs/1908.02265.
14	LI R L, YANG S, ROSS D A, et al. AI choreographer: music conditioned 3D dance generation with AIST++[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 13401-13412.
15	LIN J, YANG A, ZHANG Y, et al. Interbert: vision-and-language interaction for multi-modal pretraining[EB/OL]. [2024-02-01]. https://arxiv.org/abs/2003.13198.
16	RECASENS A, LIN J, CARREIRA J, et al. Zorro: the masked multimodal transformer[EB/OL]. [2024-02-01]. https://arxiv.org/abs/2301.09595.
17	NAGRANI A, YANG S, ARNAB A, et al. Attention bottlenecks for multimodal fusion[C]//Proceedings of NIPS'21. Cambridge, USA: MIT Press, 2021: 14200-14213.
18	WANG Y K, CHEN X H, CAO L L, et al. Multimodal token fusion for vision transformers[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, USA: IEEE Press, 2022: 12186-12195.
19	LI J N, LI D X, XIONG C M, et al. Blip: bootstrap language-image pre-training for unified vision-language understanding and generation[C]//Proceedings of the 39th International Conference on Machine Learning. Washington D. C., USA: IEEE Press, 2022: 12888-12900.
20	HUANG Z C, ZENG Z Y, LIU B, et al. Pixel-bert: aligning image pixels with text by deep multi-modal transformers[EB/OL]. [2024-02-01]. https://arxiv.org/abs/2004.00849v2.
21	SU W J, ZHU X Z, CAO Y, et al. VL-BERT: pre-training of generic visual-linguistic representations[EB/OL]. [2024-02-01]. https://arxiv.org/abs/1908:08530.
22	DESAI K R, JOHNSON J. VirTex: learning visual representations from textual annotations[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 11162-11173.
23	ZHANG Y, JIANG H, MIURA Y, et al. Contrastive learning of medical visual representations from paired images and text[EB/OL]. [2024-02-01]. http://arxiv.org/abs/2010.00747.
24	LI J N. Align before fuse: vision and language representation learning with momentum distillation[C]//Proceedings of NIPS'21. Cambridge, USA: MIT Press, 2021: 9694-9705.
25	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of International Conference on Machine Learning. Washington D. C., USA: IEEE Press, 2021: 8748-8763.
26	GAO P, GENG S J, ZHANG R R, et al. CLIP-adapter: better vision-language models with feature adapters. International Journal of Computer Vision, 2024, 132 (2): 581- 595. doi: 10.1007/s11263-023-01891-x
27	SUN Q, FANG Y, WU L, et al. Eva-clip: improved training techniques for clip at scale[EB/OL]. [2024-02-01]. http://arxiv.org/abs/2303.15389.
28	刘萌, 齐孟津, 詹圳宇, 等. 基于深度学习的图像-文本匹配研究综述. 计算机学报, 2023, 46 (11): 2370- 2399. doi: 10.11897/SP.J.1016.2023.02370
	LIU M, QI M J, ZHAN Z Y, et al. A survey on deep learning based image-text matching. Chinese Journal of Computers, 2023, 46 (11): 2370- 2399. doi: 10.11897/SP.J.1016.2023.02370
29	SHEN S, LI L H, TAN H, et al. How much can clip benefit vision-and-language tasks?[EB/OL]. [2024-02-01]. http://arxiv.org/abs/2107.06383.
30	WANG J Y, CHAN K C K, LOY C C. Exploring CLIP for assessing the look and feel of images. Artificial Intelligence, 2023, 37 (2): 2555- 2563. doi: 10.48550/arXiv.2207.12396
31	YANG J W, LI C Y, ZHANG P C, et al. Unified contrastive learning in image-text-label space[EB/OL]. [2024-02-01]. https://arxiv.org/pdf/2204.03610.pdf.
32	WANG M, ZHENG J, LIU Y. Actionclip: a new paradigm for video action recognition[EB/OL]. [2024-02-01]. http://arxiv.org/abs/2109.08472.
33	GAO Z, LIU J, CHEN S, et al. Clip2tv: an empirical study on transformer-based methods for video-text retrieval[EB/OL]. [2024-02-01]. http://arxiv.org/abs/2111.05610.
34	TANG M K, WANG Z Y, LIU Z H, et al. CLIP4Caption: clip for video caption[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York, USA: ACM Press, 2021: 4858-4862.
35	赵宏, 陈志文, 郭岚, 等. 基于ViT与语义引导的视频内容描述生成. 计算机工程, 2023, 49 (5): 247- 254. doi: 10.19678/j.issn.1000-3428.0064409
	ZHAO H, CHEN Z W, GUO L, et al. Video content caption generation based on ViT and semantic guidance. Computer Engineering, 2023, 49 (5): 247- 254. doi: 10.19678/j.issn.1000-3428.0064409
36	张颖, 张冰冰, 董微, 等. 基于语言-视觉对比学习的多模态视频行为识别方法. 自动化学报, 2024, 50 (2): 417- 430.
	ZHANG Y, ZHANG B B, DONG W, et al. Multi-modal video action recognition method based on language-visual contrastive learning. Journal of Automatica Sinica, 2024, 50 (2): 417- 430.
37	LI Y Z, ZHANG D, MU Y D. Visual-semantic matching by exploring high-order attention and distraction[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE Press, 2020: 12783-12792.

[1]	李俊仪, 李向阳, 龙朝勋, 李海燕, 李红松, 余鹏飞. 基于多级区域选择与跨层特征融合的野生菌分类[J]. 计算机工程, 2024, 50(9): 179-188.
[2]	高爽, 史轶伦, 徐巧枝, 于磊. 基于对比学习的非对称编解码结构的心脏MRI分割研究[J]. 计算机工程, 2024, 50(8): 290-300.
[3]	钱来, 赵卫伟. 基于对比学习和注意力机制的文本分类方法[J]. 计算机工程, 2024, 50(7): 104-111.
[4]	刘娟, 段友祥, 陆誉翕, 张鲁. 引入知识增强和对比学习的知识图谱补全[J]. 计算机工程, 2024, 50(7): 112-122.
[5]	武星, 殷浩宇, 姚骏峰, 李卫民, 钱权. 面向视频数据的多模态情感分析[J]. 计算机工程, 2024, 50(6): 218-227.
[6]	张宝鑫, 杨丹, 聂铁铮, 寇月. 基于自监督的多视角图协同过滤推荐方法[J]. 计算机工程, 2024, 50(5): 100-110.
[7]	李田芳, 普园媛, 赵征鹏, 徐丹, 钱文华. 基于CLIP和双空间自适应归一化的图像翻译[J]. 计算机工程, 2024, 50(5): 229-240.
[8]	张洪程, 李林育, 杨莉, 伞晨峻, 尹春林, 颜冰, 于虹, 张璇. 基于对比学习与语言模型增强嵌入的知识图谱补全[J]. 计算机工程, 2024, 50(4): 168-176.
[9]	李政学, 李枝名, 彭德中, 陈杰. 基于特征对比学习和图卷积的社交网络用户分类[J]. 计算机工程, 2024, 50(4): 258-266.
[10]	吴冠荣, 李元祥, 王艺霖, 陆雨寒, 陈秀华. 基于对比学习的小样本金属表面损伤分类[J]. 计算机工程, 2024, 50(3): 36-43.
[11]	贺子泽, 战荫伟. 基于图卷积的局部特征细化动作识别方法[J]. 计算机工程, 2024, 50(11): 276-283.
[12]	崔锦莹, 梁立河, 任雪婷, 强彦, 赵涓涓, 孔晓梅, 尉骁, 张华. 基于噪声鲁棒学习的胸部X射线尘肺鉴别方法[J]. 计算机工程, 2024, 50(11): 350-359.
[13]	曲坤, 王震龙, 刘志锋. 基于多表征学习的交叉熵集成图像分类方法[J]. 计算机工程, 2024, 50(10): 322-333.
[14]	吴永庆, 王钰涵, 朱月. 基于用户多类型反馈行为序列的点击率预估模型[J]. 计算机工程, 2024, 50(10): 405-417.
[15]	顾嘉静, 杨丹, 聂铁铮, 寇月. 基于多视图融合跨层对比学习的推荐算法[J]. 计算机工程, 2024, 50(1): 120-128.

选择文件类型/文献管理软件名称

选择包含的内容