摘要： 为提高多模态检索中相似性匹配的准确度,同时保持检索结果的可解释性,构建一种双层的多模态语义网络。对每个单模态的数据分别建立一个子语义网络,把子语义网络中的节点聚类成不同的分组。将子语义网络的分组作为节点,依据语义关系建立多模态语义网络,并进一步聚类成不同的分组。在进行信息检索时,按照与构建多模态语义网络相反的顺序即可检索到相关的信息。实验结果表明,与基于哈希索引、低秩矩阵嵌入和深度神经网络的检索方法相比,所提方法具有更高的检索准确性。

关键词: 多模态, 潜在语义, 层次模型, 聚类算法, 跨模态检索, 深度神经网络

Abstract: In order to improve the accuracy of similarity matching and ensure interpretability of retrieval results in multi-modal information retrieval,a two-layer multi-modal semantic network is proposed.Firstly,a sub-semantic network is built for the data of each single model,and the nodes in each sub-semantic network are clustered into different groups.Secondly,by assuming each group in the sub-semantic network as a node,a multi-modal semantic network is built based on semantic relationships,and the nodes in this multi-modal semantic network are further clustered into different groups.While retrieving information,the information can be retrieved by reversing steps of building the multi-modal semantic network.Experimental results show that the proposed method has higher retrieval accurary than the methods based on Hash index,low-rank matrix embedding or deep neural network.

Key words: multi-modal, latent semantic, hierarchical model, clustering algorithm, cross-modal retrieval, deep neural network

中图分类号: 

• TP319