基于特征融合的无监督跨模态哈希

doi:10.19678/j.issn.1000-3428.0063841

摘要/Abstract

摘要： 已有的无监督跨模态哈希（UCMH）方法主要关注构造相似矩阵和约束公共表征空间的结构，忽略了2个重要问题：一是它们为不同模态的数据提取独立的表征用以检索，没有考虑不同模态之间的信息互补；二是预提取特征的结构信息不完全适用于跨模态检索任务，可能会造成一些错误信息的迁移。针对第一个问题，提出一种多模态表征融合结构，通过对不同模态的嵌入特征进行融合，从而有效地综合来自不同模态的信息，提高哈希码的表达能力，同时引入跨模态生成机制，解决检索数据模态缺失的问题；针对第二个问题，提出一种相似矩阵动态调整策略，在训练过程中用学到的模态嵌入自适应地逐步优化相似矩阵，减轻预提取特征对原始数据集的偏见，使其更适应跨模态检索，并有效避免过拟合问题。基于常用数据集Flickr25k和NUS-WIDE进行实验，结果表明，通过该方法构建的模型在Flickr25k数据集上3种哈希位长检索的平均精度均值较DGCPN模型分别提高1.43%、1.82%和1.52%，在NUS-WIDE数据集上分别提高3.72%、3.77%和1.99%，验证了所提方法的有效性。

关键词: 无监督, 跨模态, 检索, 哈希, 深度学习

Abstract: Most of the Unsupervised Cross-Modal Hashing(UCMH) methods focus on the construction of a similarity matrix from the pre-extracted features and the structure control of the common representation space.However, two critical problems need to be addressed.First, the complementarity among different modalities is ignored in most studies.Second, the structural information in the pre-extracted features is partially compatible with the cross-modal retrieval task, which may cause a negative transfer.To address the first problem, this study proposes a multimodal fusion architecture.By fusing the embeddings from different modalities, the information from each modal can be integrated effectively;thus, the expressiveness of the hash codes can be improved.It also proposes a cross-modal generation mechanism to serve as an out-of-sample solution for test query data.For the second problem, this study proposes a dynamic updating strategy of the similarity matrix, which adapts it gradually with the learning embeddings in the training procedure to relieve the Bias in pre-extracted features towards the original dataset and make the similarity matrix suitable for the cross-modal retrieval task.Experiments are conducted on two widely used datasets, Flickr25k and NUS-WIDE.Consequently, the proposed method achieved improvements of 1.43%, 1.82%, and 1.52% in terms of mAP on the Flickr25k dataset with three different hash code lengths, and 3.72%, 3.77%, and 1.99% on NUS-WIDE, which demonstrates the efficacy of the proposed method.

Key words: unsupervised, cross-modal, retrieval, hashing, deep learning

中图分类号:

TP37

梁天佑, 孟敏, 武继刚. 基于特征融合的无监督跨模态哈希[J]. 计算机工程, 2023, 49(2): 90-97.

LIANG Tianyou, MENG Min, WU Jigang. Unsupervised Cross-Modal Hashing Based on Feature Fusion[J]. Computer Engineering, 2023, 49(2): 90-97.

http://www.ecice06.com/CN/Y2023/V49/I2/90

图/表 6

20230216180525

20230216180528

20230216180532

20230216180535

20230216180538

20230216180542

参考文献

[1] LIN Z J, DING G G, HU M Q, et al.Semantics-preserving hashing for cross-view retrieval[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:3864-3872.
[2] JIANG Q Y, LI W J.Discrete latent factor model for cross-modal hashing[J].IEEE Transactions on Image Processing, 2019, 28(7):3490-3501.
[3] MENG M, WANG H T, YU J, et al.Asymmetric supervised consistent and specific hashing for cross-modal retrieval[J].IEEE Transactions on Image Processing, 2021, 30:986-1000.
[4] JIANG Q Y, LI W J.Deep cross-modal hashing[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:3270-3278.
[5] LI C, DENG C, LI N, et al.Self-supervised adversarial hashing networks for cross-modal retrieval[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:4242-4251.
[6] XU R Q, LI C, YAN J C, et al.Graph convolutional network hashing for cross-modal retrieval[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence.[S.l.]:International Joint Conferences on Artificial Intelligence Organization, 2019:982-988.
[7] GU W, GU X Y, GU J Z, et al.Adversary guided asymmetric hashing for cross-modal retrieval[C]//Proceedings of 2019 International Conference on Multimedia Retrieval.New York, USA:ACM Press, 2019:159-167.
[8] ZHANG J, PENG Y X, YUAN M K.Unsupervised generative adversarial cross-modal hashing[C]//Proceedings of AAAI Conference on Artificial Intelligence.Palo Alto, USA:AAAI, 2018:1-10.
[9] SU S P, ZHONG Z H, ZHANG C.Deep joint-semantics reconstructing hashing for large-scale unsupervised cross-modal retrieval[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2019:3027-3035.
[10] HU H T, XIE L X, HONG R C, et al.Creating something from nothing:unsupervised knowledge distillation for cross-modal hashing[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:3120-3129.
[11] LIU S, QIAN S S, GUAN Y, et al.Joint-modal distribution-based similarity hashing for large-scale unsupervised deep cross-modal retrieval[C]//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.New York, USA:ACM Press, 2020:1379-1388.
[12] YANG D J, WU D Y, ZHANG W Q, et al.Deep semantic-alignment hashing for unsupervised cross-modal retrieval[C]//Proceedings of 2020 International Conference on Multimedia Retrieval.New York, USA:ACM Press, 2020:44-52.
[13] WANG W W, SHEN Y M, ZHANG H F, et al.Set and Rebase:determining the semantic graph connectivity for unsupervised cross-modal hashing[C]//Proceedings of the 29th International Joint Conference on Artificial Intelligence.[S.l.]:International Joint Conferences on Artificial Intelligence Organization, 2020:853-859.
[14] YU J, ZHOU H, ZHAN Y B, et al.Deep graph-neighbor coherence preserving network for unsupervised cross-modal hashing[C]//Proceedings of AAAI Conference on Artificial Intelligence.Palo Alto, USA:AAAI, 2021:4626-4634.
[15] DENG J, DONG W, SOCHER R, et al.ImageNet:a large-scale hierarchical image database[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2009:248-255.
[16] MIKOLOV T, CHEN K, CORRADO G, et al.Efficient estimation of word representations in vector space[EB/OL].[2022-01-25].https://arxiv.org/abs/1301.3781.
[17] KIPF T N, WELLING M.Semi-supervised classification with graph convolutional networks[EB/OL].[2022-01-25].https://arxiv.org/abs/1609.02907.
[18] VASWANI A, SHAZEER N, PARMAR N, et al.Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems.New York, USA:ACM Press, 2017:1-10.
[19] BLEI D M, NG A Y, JORDAN M I.Latent Dirichlet allocation[J].The Journal of Machine Learning Research, 2003, 3:993-1022.
[20] LE Q, MIKOLOV T.Distributed representations of sentences and documents[C]//Proceedings of the 31st International Conference on Machine Learning.New York, USA:ACM Press, 2014:1188-1196.
[21] CHEN T, KORNBLITH S, NOROUZI M, et al.A simple framework for contrastive learning of visual representations[C]//Proceedings of the 37th International Conference on Machine Learning.New York, USA:ACM Press, 2020:1597-1607.
[22] HE K M, FAN H Q, WU Y X, et al.Momentum contrast for unsupervised visual representation learning[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:9726-9735.
[23] HUISKES M J, LEW M S.The MIR flickr retrieval evaluation[C]//Proceedings of the 1st ACM International Conference on Multimedia Information Retrieval.New York, USA:ACM Press, 2008:39-43.
[24] CHUA T S, TANG J H, HONG R C, et al.NUS-WIDE:a real-world Web image database from National University of Singapore[C]//Proceedings of ACM International Conference on Image and Video Retrieval.New York, USA:ACM Press, 2009:9.
[25] SIMONYAN K, ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].[2022-01-25].https://arxiv.org/abs/1409.1556.
[26] KINGMA D P, BA J.Adam:a method for stochastic optimization[EB/OL].[2022-01-25].https://arxiv.org/abs/1412.6980.
[27] KUMAR S, UDUPA R.Learning hash functions for cross-view similarity search[C]//Proceedings of the 22nd International Joint Conference on Artificial Intelligence.Palo Alto, USA:AAAI Press, 2011:1360-1365.
[28] LIU H, JI R R, WU Y J, et al.Cross-modality binary code learning via fusion similarity hashing[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:6345-6353.
[29] DING G G, GUO Y C, ZHOU J L.Collective matrix factorization hashing for multimodal data[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2014:2083-2090.
[30] ZHOU J L, DING G G, GUO Y C.Latent semantic sparse hashing for cross-modal similarity search[C]//Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval.New York, USA:ACM Press, 2014:415-424.

选择文件类型/文献管理软件名称

选择包含的内容