基于视觉-语言预训练模型的零样本迁移学习方法综述

doi:10.19678/j.issn.1000-3428.0070036

摘要/Abstract

摘要：

近年来随着人工智能(AI)技术在计算机视觉与自然语言处理等单模态领域表现出愈发优异的性能, 多模态学习的重要性和必要性逐渐展现出来, 其中基于视觉-语言预训练模型的零样本迁移(ZST)方法得到了国内外研究者的广泛关注。得益于预训练模型强大的泛化性能, 使用视觉-语言预训练模型不仅能提高零样本识别任务的准确率, 而且能够解决部分传统方法无法解决的零样本下游任务问题。对基于视觉-语言预训练模型的ZST方法进行概述, 首先介绍了零样本学习(FSL)的传统方法, 并对其主要形式加以总结; 然后阐述了基于视觉-语言预训练模型的ZST和FSL的区别及其可以解决的新任务; 其次介绍了基于视觉-语言预训练模型的ZST方法在样本识别、目标检测、语义分割、跨模态生成等下游任务中的应用情况; 最后对现有的基于视觉-语言预训练模型的ZST方法存在的问题进行分析并对未来的研究方向进行展望。

关键词: 零样本学习, 视觉-语言预训练模型, 零样本迁移, 多模态, 计算机视觉

Abstract:

In recent years, remarkable advancements in Artificial Intelligence (AI) across unimodal domains, such as computer vision and Natural Language Processing (NLP), have highlighted the growing importance and necessity of multimodal learning. Among the emerging techniques, the Zero-Shot Transfer (ZST) method, based on visual-language pre-trained models, has garnered widespread attention from researchers worldwide. Owing to the robust generalization capabilities of pre-trained models, leveraging visual-language pre-trained models not only enhances the accuracy of zero-shot recognition tasks but also addresses certain zero-shot downstream tasks that are beyond the scope of conventional approaches. This review provides an overview of ZST methods based on vision-language pre-trained models. First, it introduces conventional approaches to Few-Shot Learning (FSL) and summarizes its main forms. It then discusses the distinctions between ZST and FSL based on vision-language pre-trained models, highlighting the new tasks that ZST can address. Subsequently, it explores the application of ZST methods in various downstream tasks, including sample recognition, object detection, semantic segmentation, and cross-modal generation. Finally, it analyzes the challenges of current ZST methods based on vision-language pre-trained models and outlines potential future research directions.

Key words: Zero-Shot Learning(ZSL), vision-language pre-trained model, Zero-Shot Transfer(ZST), multi-modal, computer vision

孙仁科, 许靖昊, 皇甫志宇, 李仲年, 许新征. 基于视觉-语言预训练模型的零样本迁移学习方法综述[J]. 计算机工程, 2024, 50(10): 1-15.

SUN Renke, XU Jinghao, HUANGFU Zhiyu, LI Zhongnian, XU Xinzheng. Survey of Zero-Shot Transfer Learning Methods Based on Vision-Language Pre-Trained Models[J]. Computer Engineering, 2024, 50(10): 1-15.

https://www.ecice06.com/CN/Y2024/V50/I10/1

图/表 7

图1 传统零样本分类方法

Fig.1 Traditional zero-shot classification methods

图2 视觉-语言预训练模型驱动的ZST流程

Fig.2 Vision-language pre-trained model-driven ZST process

图3 ZST整体架构

Fig.3 Overall architecture of ZST

参考文献 108

1	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[EB/OL]. [2024-05-11]. https://arxiv.org/abs/2103.00020.
2	JIA C, YANG Y F, XIA Y, et al. Scaling up visual and vision-language representation learning with noisy text supervision[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2102.05918.
3	ZHAI X H, WANG X, MUSTAFA B, et al. LiT: zero-shot transfer with locked-image text tuning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 18102-18112.
4	LIU H, LI C, WU Q, et al. Visual instruction tuning[C]//Proceedings of Conference on Neural Information Processing Systems (NeurIPS). Washington D.C., USA: IEEE Press, 2023: 34892-34916.
5	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2010.11929.
6	ZONG Z F, SONG G L, LIU Y. DETRs with collaborative hybrid assignments training[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2023: 6725-6735.
7	CHEN L C, ZHU Y K, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 833-851.
8	冯耀功, 于剑, 桑基韬, 等. 基于知识的零样本视觉识别综述. 软件学报, 2021, 32(2): 370- 405.
	FENG Y G, YU J, SANG J T, et al. Survey on knowledge-based zero-shot visual recognition. Journal of Software, 2021, 32(2): 370- 405.
9	PHAM H, DAI Z, GHIASI G, et al. Combined scaling for zero-shot transfer learning. Neurocomputing, 2023, 555, 126658. doi: 10.1016/j.neucom.2023.126658
10	LAMPERT C H, NICKISCH H, HARMELING S. Learning to detect unseen object classes by between-class attribute transfer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2009: 951-958.
11	PALATUCCI M, POMERLEAU D, HINTON G, et al. Zero-shot learning with semantic output codes[C]// Proceedings of the 22nd International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2009: 1410-1418.
12	ROMERA-PAREDES B, TORR P H S. An embarrassingly simple approach to zero-shot learning. Berlin, Germany: Springer International Publishing, 2017.
13	CHANGPINYO S, CHAO W L, SHA F. Predicting visual exemplars of unseen classes for zero-shot learning[C]//Proceedings of the IEEE International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2017: 3496-3505.
14	KODIROV E, XIANG T, GONG S G. Semantic autoencoder for zero-shot learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2017: 4447-4456.
15	LI Y N, WANG D H, HU H H, et al. Zero-shot recognition using dual visual-semantic mapping paths[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2017: 5207-5215.
16	刘欢, 郑庆华, 罗敏楠, 等. 基于跨域对抗学习的零样本分类. 计算机研究与发展, 2019, 56(12): 2519- 2535.
	LIU H, ZHENG Q H, LUO M N, et al. Cross-domain adversarial learning for zero-shot classification. Journal of Computer Research and Development, 2019, 56(12): 2519- 2535.
17	JIANG H J, WANG R P, SHAN S G, et al. Transferable contrastive network for generalized zero-shot learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2019: 9764-9773.
18	XIAN Y Q, SHARMA S, SCHIELE B, et al. f-VAEGAN-D2: a feature generating framework for any-shot learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2019: 10267-10276.
19	HAN Z Y, FU Z Y, CHEN S, et al. Contrastive embedding for generalized zero-shot learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2021: 2371-2381.
20	张杰, 廖盛斌, 张浩峰, 等. 基于类别扩展的广义零样本图像分类方法. 电子学报, 2023, 51(4): 1068- 1080.
	ZHANG J, LIAO S B, ZHANG H F, et al. Category expansion based generalized zero-shot image classification. Acta Electronica Sinica, 2023, 51(4): 1068- 1080.
21	WU J Z, LI X T, XU S L, et al. Towards open vocabulary learning: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(7): 5092- 5113. doi: 10.1109/TPAMI.2024.3361862
22	ALLINGHAM J U, REN J, DUSENBERRY M W, et al. A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2302.06235.
23	SHU M L, NIE W L, HUANG D A, et al. Test-time prompt tuning for zero-shot generalization in vision-language models[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2209.07511.
24	HASSAN J, GANI H, HUSSEIN N, et al. Align your prompts: test-time prompting with distribution alignment for zero-shot generalization[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2311.01459.
25	MA X, ZHANG J, GUO S, et al. SwapPrompt: test-time prompt adaptation for vision-language models[C]//Proceedings of Proceedings of the 37th International Conference on Neural Information Processing System. New York, USA: ACM Press, 2024: 65252-65264.
26	ZHANG D C, ZHOU Z, LI Y F. Robust test-time adaptation for zero-shot prompt tuning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024: 16714-16722.
27	GE Y H, REN J, GALLAGHER A, et al. Improving zero-shot generalization and robustness of multi-modal models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 11093-11101.
28	PRATT S, COVERT I, LIU R, et al. What does a platypus look like? Generating customized prompts for zero-shot image classification[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2023: 15645-15655.
29	MENON S, VONDRICK C. Visual classification via description from large language models[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2210.07183.
30	NOVACK Z, MCAULEY J, LIPTON Z C, et al. CHiLS: zero-shot image classification with hierarchical label sets[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2302.02551.
31	UDANDARAO V, GUPTA A, ALBANIE S. SuS-X: training-free name-only transfer of vision-language models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2023: 2725-2736.
32	CLARK K, JAINI P. Text-to-image diffusion models are zero-shot classifiers[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2303.15233.
33	GUO Z Y, ZHANG R R, QIU L T, et al. CALIP: zero-shot enhancement of CLIP with parameter-free attention[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2023: 746-754.
34	QIAN Q, XU Y H, HU J H. Intra-modal proxy learning for zero-shot visual categorization with CLIP[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2310.19752.
35	LI J N, SAVARESE S, HOI S C H. Masked unsupervised self-training for label-free image classification[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2206.02967.
36	LI X, BEHPOUR S, DOAN T, et al. UP-DP: unsupervised prompt learning for data pre-selection with vision-language models[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2307.11227.
37	LI X, WEN C C, HU Y, et al. RS-CLIP: zero shot remote sensing scene classification via contrastive vision-language supervision. International Journal of Applied Earth Observation and Geoinformation, 2023, 124, 103497. doi: 10.1016/j.jag.2023.103497
38	ESMAEILPOUR S, LIU B, ROBERTSON E, et al. Zero-shot out-of-distribution detection based on the pre-trained model CLIP[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2022: 6568-6576.
39	LIU K, FU Z H, CHEN C, et al. Category-extensible out-of-distribution detection via hierarchical context descriptions[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2407.16725.
40	SHIPARD J, WILIEM A, THANH K N, et al. Diversity is definitely needed: improving model-agnostic zero-shot classification via stable diffusion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Washington D.C., USA: IEEE Press, 2023: 769-778.
41	WANG Z B, LIANG J, HE R, et al. Improving zero-shot generalization for CLIP with synthesized prompts[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2307.07397.
42	LIAO N, LIU Y F, LI X B, et al. CoHOZ: contrastive multimodal prompt tuning for hierarchical open-set zero-shot recognition[C]//Proceedings of the 30th ACM International Conference on Multimedia. New York, USA: ACM Press, 2022: 3262-3271.
43	NAYAK N V, YU P L, BACH S H. Learning to compose soft prompts for compositional zero-shot learning[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2204.03574.
44	TANG B W, ZHANG J, YAN L, et al. Data-free generalized zero-shot learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024: 5108-5117.
45	KAHANA J, COHEN N, HOSHEN Y. Improving zero-shot models with label distribution priors[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2212.00784.
46	WANG H, LIU F, JIAO L C, et al. ViLT-CLIP: video and language tuning CLIP with multimodal prompt learning and scenario-guided optimization[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024: 5390-5400.
47	WANG Q, DU J L, YAN K, et al. Seeing in flowing: adapting CLIP for action recognition with motion prompts learning[C]//Proceedings of the 31st ACM International Conference on Multimedia. New York, USA: ACM Press, 2023: 5339-5347.
48	ZHENG Z W, MA M Y, WANG K, et al. Preventing zero-shot transfer degradation in continual learning of vision-language models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2023: 19068-19079.
49	ANDONIAN A, CHEN S X, HAMID R. Robust cross-modal representation learning with progressive self-distillation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 16409-16420.
50	GAO Y T, LIU J F, XU Z H, et al. PyramidCLIP: hierarchical feature alignment for vision-language model pretraining[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2204.14095.
51	YU H Y, WANG X C, LI B, et al. Chinese text recognition with A pre-trained CLIP-like model through image-IDS aligning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2023: 11909-11918.
52	LU M Y, CHEN B W, ZHANG A, et al. Visual language pretrained multiple instance zero-shot transfer for histopathology images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 19764-19775.
53	WANG Z F, WU Z B, AGARWAL D, et al. MedCLIP: contrastive learning from unpaired medical images and text[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: Association for Computational Linguistics, 2022: 3876-3887.
54	LIN Y Q, CHEN M H, ZHANG K P, et al. TagCLIP: a local-to-global framework to enhance open-vocabulary multi-label classification of CLIP without training[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024: 3513-3521.
55	GU X Y, LIN T Y, KUO W C, et al. Open-vocabulary object detection via vision and language knowledge distillation[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2104.13921.
56	DU Y, WEI F Y, ZHANG Z H, et al. Learning to prompt for open-vocabulary object detection with vision-language model[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 14064-14073.
57	LIU Z M, HU X F, NEVATIA R. Efficient feature distillation for zero-shot annotation object detection[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Washington D.C., USA: IEEE Press, 2024: 882-891.
58	MA Z Y, LUO G, GAO J, et al. Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 14054-14063.
59	RASHEED H, MAAZ M, KHATTAK M U, et al. Bridging the gap between object and image-level representations for open-vocabulary detection[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2207.03482.
60	WANG L T, LIU Y, DU P H, et al. Object-aware distillation pyramid for open-vocabulary object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 11186-11196.
61	ZHONG Y W, YANG J W, ZHANG P C, et al. RegionCLIP: region-based language-image pretraining[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 16772-16782.
62	GAO M F, XING C, NIEBLES J C, et al. Open vocabulary object detection with pseudo bounding-box labels[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2111.09452.
63	ZHAO S Y, ZHANG Z X, SCHULTER S, et al. Exploiting unlabeled data with vision and language models for object detection. Berlin, Germany: Springer, 2022.
64	LONG Y, HAN J, HUANG R, et al. Fine-grained visual-text prompt-driven self-training for open-vocabulary object detection [EB/OL]. [2024-05-11]. http://export.arxiv.org/abs/2211.00849
65	MINDERER M, GRITSENKO A, HOULSBY N. Scaling open-vocabulary object detection[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2306.09683.
66	FENG C J, ZHONG Y J, JIE Z Q, et al. PromptDet: towards open-vocabulary detection using uncurated images[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2022: 701-717.
67	WU X S, ZHU F, ZHAO R, et al. CORA: adapting CLIP for open-vocabulary detection with region prompting and anchor pre-matching[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 7031-7040.
68	FANG R H, PANG G S, BAI X. Simple image-level classification improves open-vocabulary object detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024: 1716-1725.
69	KUO W C, CUI Y, GU X Y, et al. F-VLM: open-vocabulary object detection upon frozen vision and language models[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2209.15639.
70	JEONG J, PARK G, YOO J, et al. ProxyDet: synthesizing proxy novel classes via classwise mixup for open-vocabulary object detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024: 2462-2470.
71	LIN J H, SHEN Y H, WANG B Q, et al. Weakly supervised open-vocabulary object detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024: 3404-3412.
72	SHEN H Z, ZHAO T C, ZHU M W, et al. GroundVLP: harnessing zero-shot visual grounding from vision-language pre-training and open-vocabulary object detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024: 4766-4775.
73	YAO L W, HAN J H, WEN Y P, et al. DetCLIP: dictionary-enriched visual-concept paralleled pre-training for open-world detection[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2209.09407.
74	ZHOU X Y, GIRDHAR R, JOULIN A, et al. Detecting twenty-thousand classes using image-level supervision. Berlin, Germany: Springer, 2022.
75	WANG T. Learning to detect and segment for open vocabulary object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 7051-7060.
76	XIE J, ZHENG S. Zero-shot object detection through vision-language embedding alignment[C]//Proceedings of the IEEE International Conference on Data Mining Workshops (ICDMW). Washington D.C., USA: IEEE Press, 2022: 1-15.
77	ZHANG D M, LI C, ZHANG R R, et al. FM-OV3D: foundation model-based cross-modal knowledge blending for open-vocabulary 3D detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024: 16723-16731.
78	MAO Y, DENG J, ZHOU W, et al. CLIP4HOI: towards adapting CLIP for practical zero-shot HOI detection[C]//Proceedings of the 37th International Conference on Neural Information Processing System. New York, USA: ACM Press, 2024: 45895-45906.
79	NAG S, ZHU X T, SONG Y Z, et al. Zero-shot temporal action detection via vision-language prompting[C]// Proceedings of the European Conference on Computer Vision. Berlin, Germany: Springer, 2022: 681-697.
80	LI L, XIAO J, CHEN G K, et al. Zero-shot visual relation detection via composite visual cues from large language models[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2305.12476.
81	YANG S, WANG Y Q, JI X F, et al. Multi-modal prompting for open-vocabulary video visual relationship detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024: 6513-6521.
82	XU W H, XU R T, WANG C W, et al. Spectral prompt tuning: unveiling unseen classes for zero-shot semantic segmentation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024: 6369-6377.
83	MA C F, YANG Y H, WANG Y F, et al. Open-vocabulary semantic segmentation with frozen vision-language models[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2210.15138.
84	JEONG J, ZOU Y, KIM T, et al. WinCLIP: zero-/few-shot anomaly classification and segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 19606-19616.
85	LI B Y, WEINBERGER K Q, BELONGIE S, et al. Language-driven semantic segmentation[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2201.03546.
86	DING J, XUE N, XIA G S, et al. Decoupling zero-shot semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 11573-11582.
87	SHIN G, XIE W D, ALBANIE S. ReCo: retrieve and co-segment for zero-shot transfer[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2206.07045.
88	GHIASI G, GU X Y, CUI Y, et al. Scaling open-vocabulary image segmentation with image-level labels. Berlin, Germany: Springer, 2022.
89	PANDEY P, CHASMAI M, NATARAJAN M, et al. A language-guided benchmark for weakly supervised open vocabulary semantic segmentation[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2302.14163.
90	LIANG F, WU B C, DAI X L, et al. Open-vocabulary semantic segmentation with mask-adapted CLIP[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 7061-7070.
91	YU Q H, HE J, DENG X Q, et al. Convolutions die hard: open-vocabulary segmentation with single frozen convolutional CLIP[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2308.02487.
92	FAHES M, VU T H, BURSUC A, et al. PØDA: prompt-driven zero-shot domain adaptation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2023: 18577-18587.
93	YANG L F, WANG Y Z, LI X, et al. Fine-grained visual prompting[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2306.04356.
94	ZHOU C, LOY C C, DAI B. Extract free dense labels from CLIP. Berlin, Germany: Springer, 2022.
95	JIAO S Y, WEI Y C, WANG Y W, et al. Learning mask-aware CLIP representations for zero-shot segmentation[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2310.00240.
96	ZHOU Z Q, LEI Y J, ZHANG B W, et al. ZegCLIP: towards adapting CLIP for zero-shot semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 11175-11185.
97	YU S, SEO P H, SON J. Zero-shot referring image segmentation with global-local context features[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 19456-19465.
98	WANG Y B, HUANG S F, GAO Y L, et al. Transferring CLIP's knowledge into zero-shot point cloud semantic segmentation[C]//Proceedings of the 31st ACM International Conference on Multimedia. New York, USA: ACM Press, 2023: 3745-3754.
99	LIU K H, ZHAN F N, ZHANG J H, et al. Weakly supervised 3D open-vocabulary segmentation[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2305.14093.
100	LI W, ZHU L C, WEN L Y, et al. DeCap: decoding CLIP latents for zero-shot captioning via text-only training[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2303.03032.
101	QIU L T, NING S, HE X M. Mining fine-grained image-text alignment for zero-shot captioning via text-only training[EB/OL]. [2024-05-11]. http://arxiv.org/abs/2401.02347.
102	FEI J J, WANG T, ZHANG J R, et al. Transferable decoding with visual entities for zero-shot image captioning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2023: 3113-3123.
103	刘天义, 吴祖煊, 陈静静, 等. 面向视觉语言理解与生成的多模态预训练方法. 软件学报, 2022, 34(5): 2024- 2034.
	LIU T Y, WU Z X, CHEN J J, et al. Multimodal pre-training method for vision-language understanding and generation. Journal of Software, 2023, 34(5): 2024- 2034.
104	GUO J Y, WANG C F, WU Y, et al. Zero-shot generative model adaptation via image-specific prompt learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 11494-11503.
105	BALDRATI A, AGNOLUCCI L, BERTINI M, et al. Zero-shot composed image retrieval with textual inversion[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2023: 15292-15301.
106	SAIN A, BHUNIA A K, CHOWDHURY P N, et al. CLIP for all things zero-shot sketch-based image retrieval, fine-grained or not[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 2765-2775.
107	JIANG R X, LIU L B, CHEN C W. CLIP-Count: towards text-guided zero-shot object counting[C]//Proceedings of the 31st ACM International Conference on Multimedia. New York, USA: ACM Press, 2023: 4535-4545.
108	LUO J Q, WANG Z N, WU C H, et al. Zero-shot model diagnosis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 11631-11640.

[1]	李华昱, 张智康, 闫阳, 岳阳. 基于知识图谱增强的领域多模态实体识别[J]. 计算机工程, 2024, 50(8): 31-39.
[2]	刘建敏, 林晖, 汪晓丁. 基于图注意力机制的无地图场景轨迹预测方法[J]. 计算机工程, 2024, 50(7): 144-153.
[3]	武星, 殷浩宇, 姚骏峰, 李卫民, 钱权. 面向视频数据的多模态情感分析[J]. 计算机工程, 2024, 50(6): 218-227.
[4]	董峰, 王永欣, 马玉玲, 王奎奎. 原型对齐和域感知的零样本哈希[J]. 计算机工程, 2024, 50(5): 260-271.
[5]	吴海鹏, 钱育蓉, 冷洪勇. 基于双向注意力机制的多模态关系抽取[J]. 计算机工程, 2024, 50(4): 160-167.
[6]	连哲, 殷雁君, 云飞, 智敏. 基于深度学习的自然场景文本检测综述[J]. 计算机工程, 2024, 50(3): 16-27.
[7]	朱贵德, 黄海. 文本视觉问答综述[J]. 计算机工程, 2024, 50(2): 1-14.
[8]	顾宇衡, 潘嘉诚, 钱江波, 董一鸿. 阿尔茨海默病的图神经网络分类方法研究进展[J]. 计算机工程, 2024, 50(10): 35-50.
[9]	吴志强, 解庆, 李琳, 刘永坚. 基于多模态融合的图神经网络推荐算法[J]. 计算机工程, 2024, 50(1): 91-100.
[10]	周昊玮, 刘勇, 玄萍. 基于预训练和多模态融合的假新闻检测[J]. 计算机工程, 2024, 50(1): 289-295.
[11]	曹广硕, 黄瑞章, 陈艳平, 秦永彬. 基于多模态学习的乳腺癌生存预测研究[J]. 计算机工程, 2024, 50(1): 296-305.
[12]	洪犇, 钱旭升, 申明磊, 胡冀苏, 耿辰, 戴亚康, 周志勇. 基于深度学习的CT-MR图像联合配准分割方法[J]. 计算机工程, 2023, 49(9): 234-245.
[13]	崔晓丹, 刘达维, 刘逸凡, 赵志滨, 任酉贵, 闫永明. 新闻类短视频关键帧摘要模型的研究与实现[J]. 计算机工程, 2023, 49(8): 182-189.
[14]	程适, 王雪萍, 刘悦, 史玉回. 面向非线性方程组的学习型头脑风暴优化算法[J]. 计算机工程, 2023, 49(7): 47-54.
[15]	郭艳霞, 金勇, 唐宏, 彭金枝. 基于动态卷积与残差门控的多模态情感识别[J]. 计算机工程, 2023, 49(7): 94-101.

选择文件类型/文献管理软件名称

选择包含的内容