基于深度互学习的多标记零样本分类

doi:10.19678/j.issn.1000-3428.0065806

摘要/Abstract

摘要：

目前已有大量方案解决零样本图像分类问题，但对多标记零样本图像分类问题的研究很少，在现有的解决方案中，模型在训练时除了利用已标注的数据集和给定的先验知识外，只利用图像区域信息或只利用标签语义信息。基于深度互学习技术，提出一种能同时利用图像区域和标签语义两种信息的解决方法。设计两个子网络，将子网络1用于增强图像视觉特征，通过多头自注意机制关联图像中不同区域的特征信息，得到基于区域的视觉特征表示，再将该特征表示映射到语义空间中，并输出预测概率分布；使子网络2用于融合标签语义信息与图像视觉特征，通过计算标签和图像区域特征的相关性，得到基于语义的视觉特征表示，将特征表示映射到语义空间中输出概率分布。最后引入深度互学习技术，利用两个子网络得到的概率分布为对方提供训练经验以进行互相学习，该过程中子网络在训练自身分类性能的同时也学习对方的训练经验，有效提升多标记零样本图像分类的性能。实验结果表明，所提方法在MS COCO数据集上的F1值相比Deep0Tag方法提升了5.2个百分点。

关键词: 深度学习, 图像分类, 多标记学习, 零样本学习, 互学习

Abstract:

Numerous methods have been proposed to solve the zero-shot image classification problem; however, there are limited studies on the multi-label zero-shot image classification problem. In the existing solutions, in addition to the use of the basic settings of the labeled dataset and the given prior knowledge, the model either only uses the image region information or only the label semantic information. Based on deep mutual learning technology, this study proposes a solution that utilizes both the image region and label semantic information. Two sub-networks are designed. Sub-network 1 is used to enhance the visual features of the image, whereas the multi-head self-attention mechanism is used to associate the feature information of different regions in the image to obtain a region-based visual feature representation and then map the feature representation to the semantic space to output the predicted probability. Sub-network 2 is used to fuse the label semantic information and image visual features by calculating the correlation between the labels and image region features to obtain a semantic-based visual feature representation, and then map the feature representation to the semantic space to output a probability distribution. Finally, the deep mutual learning technology is introduced, and the probability distribution obtained by the two sub-networks is used to provide training experience for mutual learning. In this process, the sub-network refers to the training experience of the other sub-network while training its own classification performance, which effectively improves the performance of multi-label zero-shot image classification. The experimental results show that the F1 value of the proposed method on the MS COCO dataset increased by 5.2 percentage points compared to the Deep0Tag method.

Key words: deep learning, image classification, multi-label learning, zero-shot learning, mutual learning

袁志祥, 王雅卿, 黄俊. 基于深度互学习的多标记零样本分类[J]. 计算机工程, 2023, 49(10): 64-71.

Zhixiang YUAN, Yaqing WANG, Jun HUANG. Multi-Label Zero-Shot Classification Based on Deep Mutual Learning[J]. Computer Engineering, 2023, 49(10): 64-71.

https://www.ecice06.com/CN/Y2023/V49/I10/64

图/表 8

图1 基于深度互学习的多标记零样本学习模型

Fig.1 Multi-label zero-shot learning model based on deep mutual learning

图2 不同互学习损失系数λ对模型性能的影响

Fig.2 Effect of different mutual learning loss coefficients λ on model performance

图3 NUS-WIDE数据集上不同权重系数 α 对模型性能的影响

Fig.3 Effect of different weight coefficients α on model performance on NUS-WIDE data set

图4 MS COCO数据集上不同权重系数 α 对模型性能的影响

Fig.4 Effect of different weight coefficients α on model performance on MS COCO data set

参考文献 28

1	BOUTELL M R, LUO J B, SHEN X P, et al. Learning multi-label scene classification. Pattern Recognition, 2004, 37(9): 1757- 1771. doi: 10.1016/j.patcog.2004.03.009
2	蔡亚萍, 杨明. 一种利用局部标记相关性的多标记特征选择算法. 南京大学学报(自然科学), 2016, 52(4): 693- 704. URL
	CAI Y P, YANG M. A multi-label feature selection algorithm by exploiting label correlations locally. Journal of Nanjing University (Natural Science), 2016, 52(4): 693- 704. URL
3	朱赛赛, 贾修一, 李泽超. 一种基于全局和局部标记相关性的多标记分类算法. 电子学报, 2020, 48(12): 2345- 2351. URL
	ZHU S S, JIA X Y, LI Z C. Exploiting global and loca label correlations for multi-label classification. Acta Electronica Sinica, 2020, 48(12): 2345- 2351. URL
4	KIPF T, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. [2022-08-10]. https://arxiv.org/pdf/1609.02907.pdf.
5	CHEN Z M, WEI X S, WANG P, et al. Multi-label image recognition with graph convolutional networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 5172-5181.
6	CHEN T S, XU M X, HUI X L, et al. Learning semantic-specific graph representation for multi-label image recognition[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2020: 522-531.
7	WANG Z X, CHEN T S, LI G B, et al. Multi-label image recognition by recurrently discovering attentional regions[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 464-472.
8	YOU R C, GUO Z Y, CUI L, et al. Cross-modality attention with semantic graph embedding for multi-label classification. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 12709- 12716. doi: 10.1609/aaai.v34i07.6964
9	LIXIA X, DI J, RONGGUI W, et al. Multi-label classification based on attention mechanism and semantic dependencies[EB/OL]. [2022-08-10]. http://en.cnki.com.cn/Article_en/CJFDTotal-GDGC201909003.html.
10	AKATA Z, PERRONNIN F, HARCHAOUI Z, et al. Label-embedding for attribute-based classification[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2013: 819-826.
11	LI Y, ZHANG J G, ZHANG J G, et al. Discriminative learning of latent features for zero-shot recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7463-7471.
12	魏宏喜, 张越. 基于生成对抗网络的零样本图像分类. 北京航空航天大学学报, 2019, 45(12): 2345- 2350. URL
	WEI H X, ZHANG Y. Zero-shot image classification based on generative adversarial network. Journal of Beijing University of Aeronautics and Astronautics, 2019, 45(12): 2345- 2350. URL
13	LI Y, SWERSKY K, ZEMEL R. Generative moment matching networks[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1502.02761.
14	LEE C W, FANG W, YEH C K, et al. Multi-label zero-shot learning with structured knowledge graphs[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 1576-1585.
15	GOA B, GYAB C, CD D, et al. Multi-label zero-shot learning with graph convolutional networks. Neural Networks, 2020, 132, 333- 341. doi: 10.1016/j.neunet.2020.09.010
16	GUPTA A, NARAYAN S, KHAN S, et al. Generative multi-label zero-shot learning[EB/OL]. [2022-08-10]. https://arxiv.org/abs/2101.11606.
17	KIM J H, JUN J, ZHANG B T. Bilinear attention networks[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1805.07932.
18	HUYNH D, ELHAMIFAR E. A shared multi-attention framework for multi-label zero-shot learning[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 8773-8783.
19	NARAYAN S, GUPTA A, KHAN S, et al. Discriminative region-based multi-label zero-shot learning[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2022: 8711-8720.
20	ZHANG Y, XIANG T, HOSPEDALES T M, et al. Deep mutual learning[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 4320-4328.
21	CHUA T S, TANG J H, HONG R C, et al. NUS-WIDE: a real-world web image database from National University of Singapore[C]//Proceedings of the ACM International Conference on Image and Video Retrieval. New York, USA: ACM Press, 2009: 1-9.
22	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]///Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 740-755.
23	BEN-COHEN A, ZAMIR N, BEN BARUCH E, et al. Semantic diversity learning for zero-shot multi-label classification[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2022: 620-630.
24	ZHANG Y, GONG B Q, SHAH M. Fast zero-shot image tagging[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 5985-5994.
25	PENNINGTON J, SOCHER R, MANNING C. Glove: global vectors for word representation[C]//Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: Association for Computational Linguistics, 2014: 1532-1543.
26	NOROUZI M, MIKOLOV T, BENGIO S, et al. Zero-shot learning by convex combination of semantic embeddings[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1312.5650.
27	AKATA Z, PERRONNIN F, HARCHAOUI Z, et al. Label-embedding for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(7): 1425- 1438.
28	RAHMAN S, KHAN S, BARNES N. Deep0Tag: deep multiple instance learning for zero-shot image tagging. IEEE Transactions on Multimedia, 2020, 22(1): 242- 255.

[1]	魏嵬, 丁香香, 郭梦星, 杨钊, 刘辉. 文本相似度计算方法综述[J]. 计算机工程, 2024, 50(9): 18-32.
[2]	李俊仪, 李向阳, 龙朝勋, 李海燕, 李红松, 余鹏飞. 基于多级区域选择与跨层特征融合的野生菌分类[J]. 计算机工程, 2024, 50(9): 179-188.
[3]	朱凯, 李理, 张彤, 江晟, 别一鸣. 基于Transformer的多阶段运动模糊图像修复网络[J]. 计算机工程, 2024, 50(9): 276-285.
[4]	张天鹏, 韩晶, 吕学强. 基于多任务学习的超分辨率辅助小目标检测[J]. 计算机工程, 2024, 50(9): 304-312.
[5]	高煜宝, 文志诚. 基于注意力机制的双路解码器图像去噪方法[J]. 计算机工程, 2024, 50(9): 324-332.
[6]	张华青, 夏张涛, 陆晓庆, 童基均. 基于字形特征的血管外科命名实体识别[J]. 计算机工程, 2024, 50(8): 13-21.
[7]	张亚洲, 和玉, 戎璐, 王祥凯. 基于上下文知识增强型Transformer网络的抑郁检测[J]. 计算机工程, 2024, 50(8): 75-85.
[8]	高伟, 李帅龙, 茆琳, 王磊, 李颖颖, 韩林. 一种基于TVM的算子生成加速策略[J]. 计算机工程, 2024, 50(8): 353-362.
[9]	王宇, 祁琦, 王纯, 许才. 储能变流器信号高精度故障诊断方法[J]. 计算机工程, 2024, 50(8): 389-396.
[10]	牛瑞婷, 严天峰, 高锐, 王映植. 低信噪比下基于深度学习TCNN-MobileNet的调制识别[J]. 计算机工程, 2024, 50(7): 204-215.
[11]	肖慈, 徐杨, 张永丹, 冯明文, 黄易仟. 结合注意力和低光增强的夜间语义分割[J]. 计算机工程, 2024, 50(7): 271-281.
[12]	张诗婧, 莫绪涛, 赵行, 董杨林. 基于球面折反射成像和YOLOv7的内螺纹缺陷检测[J]. 计算机工程, 2024, 50(7): 282-292.
[13]	徐明亮, 李芳媛, 马浩然, 何飞. 大规模神经记录的峰电位聚类算法(特邀)[J]. 计算机工程, 2024, 50(6): 1-34.
[14]	魏琢艺, 罗迈, 李文兵, 曾远松, 余伟江, 杨跃东. 基于多源域适应的单细胞智能分类[J]. 计算机工程, 2024, 50(6): 48-55.
[15]	李子杰, 周菊香, 韩晓瑜, 甘健侯, 鹿泽光, 王俊. 序列特征与学习过程融合的知识追踪模型[J]. 计算机工程, 2024, 50(6): 77-85.

选择文件类型/文献管理软件名称

选择包含的内容