基于深度互学习的多标记零样本分类

doi:10.19678/j.issn.1000-3428.0065806

摘要/Abstract

摘要：

目前已有大量方案解决零样本图像分类问题，但对多标记零样本图像分类问题的研究很少，在现有的解决方案中，模型在训练时除了利用已标注的数据集和给定的先验知识外，只利用图像区域信息或只利用标签语义信息。基于深度互学习技术，提出一种能同时利用图像区域和标签语义两种信息的解决方法。设计两个子网络，将子网络1用于增强图像视觉特征，通过多头自注意机制关联图像中不同区域的特征信息，得到基于区域的视觉特征表示，再将该特征表示映射到语义空间中，并输出预测概率分布；使子网络2用于融合标签语义信息与图像视觉特征，通过计算标签和图像区域特征的相关性，得到基于语义的视觉特征表示，将特征表示映射到语义空间中输出概率分布。最后引入深度互学习技术，利用两个子网络得到的概率分布为对方提供训练经验以进行互相学习，该过程中子网络在训练自身分类性能的同时也学习对方的训练经验，有效提升多标记零样本图像分类的性能。实验结果表明，所提方法在MS COCO数据集上的F1值相比Deep0Tag方法提升了5.2个百分点。

关键词: 深度学习, 图像分类, 多标记学习, 零样本学习, 互学习

Abstract:

Numerous methods have been proposed to solve the zero-shot image classification problem; however, there are limited studies on the multi-label zero-shot image classification problem. In the existing solutions, in addition to the use of the basic settings of the labeled dataset and the given prior knowledge, the model either only uses the image region information or only the label semantic information. Based on deep mutual learning technology, this study proposes a solution that utilizes both the image region and label semantic information. Two sub-networks are designed. Sub-network 1 is used to enhance the visual features of the image, whereas the multi-head self-attention mechanism is used to associate the feature information of different regions in the image to obtain a region-based visual feature representation and then map the feature representation to the semantic space to output the predicted probability. Sub-network 2 is used to fuse the label semantic information and image visual features by calculating the correlation between the labels and image region features to obtain a semantic-based visual feature representation, and then map the feature representation to the semantic space to output a probability distribution. Finally, the deep mutual learning technology is introduced, and the probability distribution obtained by the two sub-networks is used to provide training experience for mutual learning. In this process, the sub-network refers to the training experience of the other sub-network while training its own classification performance, which effectively improves the performance of multi-label zero-shot image classification. The experimental results show that the F1 value of the proposed method on the MS COCO dataset increased by 5.2 percentage points compared to the Deep0Tag method.

Key words: deep learning, image classification, multi-label learning, zero-shot learning, mutual learning

袁志祥, 王雅卿, 黄俊. 基于深度互学习的多标记零样本分类[J]. 计算机工程, 2023, 49(10): 64-71.

Zhixiang YUAN, Yaqing WANG, Jun HUANG. Multi-Label Zero-Shot Classification Based on Deep Mutual Learning[J]. Computer Engineering, 2023, 49(10): 64-71.

http://www.ecice06.com/CN/Y2023/V49/I10/64

图/表 8

图1 基于深度互学习的多标记零样本学习模型

Fig.1 Multi-label zero-shot learning model based on deep mutual learning

图2 不同互学习损失系数λ对模型性能的影响

Fig.2 Effect of different mutual learning loss coefficients λ on model performance

图3 NUS-WIDE数据集上不同权重系数 α 对模型性能的影响

Fig.3 Effect of different weight coefficients α on model performance on NUS-WIDE data set

图4 MS COCO数据集上不同权重系数 α 对模型性能的影响

Fig.4 Effect of different weight coefficients α on model performance on MS COCO data set

参考文献 28

1	BOUTELL M R, LUO J B, SHEN X P, et al. Learning multi-label scene classification. Pattern Recognition, 2004, 37(9): 1757- 1771. doi: 10.1016/j.patcog.2004.03.009
2	蔡亚萍, 杨明. 一种利用局部标记相关性的多标记特征选择算法. 南京大学学报(自然科学), 2016, 52(4): 693- 704. URL
	CAI Y P, YANG M. A multi-label feature selection algorithm by exploiting label correlations locally. Journal of Nanjing University (Natural Science), 2016, 52(4): 693- 704. URL
3	朱赛赛, 贾修一, 李泽超. 一种基于全局和局部标记相关性的多标记分类算法. 电子学报, 2020, 48(12): 2345- 2351. URL
	ZHU S S, JIA X Y, LI Z C. Exploiting global and loca label correlations for multi-label classification. Acta Electronica Sinica, 2020, 48(12): 2345- 2351. URL
4	KIPF T, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. [2022-08-10]. https://arxiv.org/pdf/1609.02907.pdf.
5	CHEN Z M, WEI X S, WANG P, et al. Multi-label image recognition with graph convolutional networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 5172-5181.
6	CHEN T S, XU M X, HUI X L, et al. Learning semantic-specific graph representation for multi-label image recognition[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2020: 522-531.
7	WANG Z X, CHEN T S, LI G B, et al. Multi-label image recognition by recurrently discovering attentional regions[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 464-472.
8	YOU R C, GUO Z Y, CUI L, et al. Cross-modality attention with semantic graph embedding for multi-label classification. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 12709- 12716. doi: 10.1609/aaai.v34i07.6964
9	LIXIA X, DI J, RONGGUI W, et al. Multi-label classification based on attention mechanism and semantic dependencies[EB/OL]. [2022-08-10]. http://en.cnki.com.cn/Article_en/CJFDTotal-GDGC201909003.html.
10	AKATA Z, PERRONNIN F, HARCHAOUI Z, et al. Label-embedding for attribute-based classification[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2013: 819-826.
11	LI Y, ZHANG J G, ZHANG J G, et al. Discriminative learning of latent features for zero-shot recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7463-7471.
12	魏宏喜, 张越. 基于生成对抗网络的零样本图像分类. 北京航空航天大学学报, 2019, 45(12): 2345- 2350. URL
	WEI H X, ZHANG Y. Zero-shot image classification based on generative adversarial network. Journal of Beijing University of Aeronautics and Astronautics, 2019, 45(12): 2345- 2350. URL
13	LI Y, SWERSKY K, ZEMEL R. Generative moment matching networks[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1502.02761.
14	LEE C W, FANG W, YEH C K, et al. Multi-label zero-shot learning with structured knowledge graphs[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 1576-1585.
15	GOA B, GYAB C, CD D, et al. Multi-label zero-shot learning with graph convolutional networks. Neural Networks, 2020, 132, 333- 341. doi: 10.1016/j.neunet.2020.09.010
16	GUPTA A, NARAYAN S, KHAN S, et al. Generative multi-label zero-shot learning[EB/OL]. [2022-08-10]. https://arxiv.org/abs/2101.11606.
17	KIM J H, JUN J, ZHANG B T. Bilinear attention networks[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1805.07932.
18	HUYNH D, ELHAMIFAR E. A shared multi-attention framework for multi-label zero-shot learning[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 8773-8783.
19	NARAYAN S, GUPTA A, KHAN S, et al. Discriminative region-based multi-label zero-shot learning[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2022: 8711-8720.
20	ZHANG Y, XIANG T, HOSPEDALES T M, et al. Deep mutual learning[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 4320-4328.
21	CHUA T S, TANG J H, HONG R C, et al. NUS-WIDE: a real-world web image database from National University of Singapore[C]//Proceedings of the ACM International Conference on Image and Video Retrieval. New York, USA: ACM Press, 2009: 1-9.
22	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]///Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 740-755.
23	BEN-COHEN A, ZAMIR N, BEN BARUCH E, et al. Semantic diversity learning for zero-shot multi-label classification[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2022: 620-630.
24	ZHANG Y, GONG B Q, SHAH M. Fast zero-shot image tagging[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 5985-5994.
25	PENNINGTON J, SOCHER R, MANNING C. Glove: global vectors for word representation[C]//Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: Association for Computational Linguistics, 2014: 1532-1543.
26	NOROUZI M, MIKOLOV T, BENGIO S, et al. Zero-shot learning by convex combination of semantic embeddings[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1312.5650.
27	AKATA Z, PERRONNIN F, HARCHAOUI Z, et al. Label-embedding for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(7): 1425- 1438.
28	RAHMAN S, KHAN S, BARNES N. Deep0Tag: deep multiple instance learning for zero-shot image tagging. IEEE Transactions on Multimedia, 2020, 22(1): 242- 255.

[1]	池亚平, 岳梓岩, 林雨衡. 基于Transformer的SM4算法工作模式识别[J]. 计算机工程, 2023, 49(9): 109-117.
[2]	林中霖, 时金桥, 王美琪, 王学宾, 王雨燕. 基于应用行为划分的Android恶意应用检测技术[J]. 计算机工程, 2023, 49(9): 125-136.
[3]	江雨燕, 陶承凤, 李平. 数据增强和自适应自步学习的深度子空间聚类算法[J]. 计算机工程, 2023, 49(8): 96-103, 110.
[4]	李泽水, 冀俊忠, 杨翠翠. 基于边权重信息深度网络嵌入的PPIN功能模块检测[J]. 计算机工程, 2023, 49(8): 69-76.
[5]	王可铮, 徐玉芬, 周尚波. 结合对比感知损失和融合注意力的图像去雾模型[J]. 计算机工程, 2023, 49(8): 207-214.
[6]	刘俊豪, 王美林, 谢兴, 宋烨兴, 许莉花. 基于改进YOLOv5的皮革瑕疵检测算法[J]. 计算机工程, 2023, 49(8): 240-249.
[7]	曹坪, 杨怀志, 薄一军, 尤嘉, 张淳杰, 李丹勇. 面向低质量裂缝图像的多知识蒸馏分类[J]. 计算机工程, 2023, 49(7): 204-213.
[8]	闫兴亚, 匡娅茜, 白光睿, 李月. 基于深度学习的学生课堂行为识别方法[J]. 计算机工程, 2023, 49(7): 251-258.
[9]	李军侠, 王星驰, 殷梓, 石德硕. 边缘深度挖掘的弱监督显著性目标检测[J]. 计算机工程, 2023, 49(7): 169-178.
[10]	吴珊, 周凤. 基于改进SSD算法的小目标检测[J]. 计算机工程, 2023, 49(7): 179-188.
[11]	席建锐, 唐红梅, 梁春阳, 刘鑫. 基于改进隐函数的点云物体重建[J]. 计算机工程, 2023, 49(7): 214-222.
[12]	齐咏生, 杜晓旭, 朱俊峰, 高胜利, 刘利强. 基于增强型轻量深度网络的牧区牲畜高效检测[J]. 计算机工程, 2023, 49(7): 278-287.
[13]	谌雨章, 黄逸姿, 张钧涵. 基于多速率空洞卷积的多尺度水下小目标检测[J]. 计算机工程, 2023, 49(6): 257-264.
[14]	张博旭, 蒲智, 程曦. 基于提示学习的维吾尔语文本分类研究[J]. 计算机工程, 2023, 49(6): 292-299,313.
[15]	于海洋, 景鹏, 张文涛, 谢赛飞, 滑志华, 宋草原. 基于残差与注意力机制的道路裂缝检测U-Net改进模型[J]. 计算机工程, 2023, 49(6): 265-273.

选择文件类型/文献管理软件名称

选择包含的内容