基于多模态模式迁移的知识图谱实体配图

doi:10.19678/j.issn.1000-3428.0064039

计算机工程 ›› 2022, Vol. 48 ›› Issue (8): 70-76. doi: 10.19678/j.issn.1000-3428.0064039

基于多模态模式迁移的知识图谱实体配图

蒋雪瑶¹, 力维辰¹, 刘井平², 李直旭¹, 肖仰华¹

1. 复旦大学软件学院, 上海 200433;
2. 华东理工大学信息科学与工程学院, 上海 200237

收稿日期:2022-02-25 修回日期:2022-04-05 发布日期:2022-05-25
作者简介:蒋雪瑶(1997-),女,硕士研究生,主研方向为多模态知识图谱;力维辰,硕士研究生;刘井平,讲师、博士;李直旭、肖仰华,教授、博士。
基金资助:
上海市科技创新行动计划（19511120400）。

Entity Image Collection Based on Multi-Modality Pattern Transfer

JIANG Xueyao¹, LI Weichen¹, LIU Jingping², LI Zhixu¹, XIAO Yanghua¹

1. School of Software, Fudan University, Shanghai 200433, China;
2. School of Information Science and Engineering, East China University of Science and Technology, Shanghai 200237, China

Received:2022-02-25 Revised:2022-04-05 Published:2022-05-25

摘要/Abstract

摘要： 构建多模态知识图谱的核心在于为知识图谱中的实体匹配正确合适的图像。现有的实体配图方法主要将百科图谱以及图像搜索引擎作为实体候选图像的来源，但对图像数据元的应用方式比较简单，不能准确把握图像数据来源的特点，且可扩展性较差。提出一种基于多模态模式迁移的知识图谱实体配图方法，从不同类别的头部实体中抽取对应的语义模板及视觉模式迁移到同类非头部实体的图像获取过程中，其中语义模板用于构建搜索引擎检索关键词，视觉模式用于对检索结果去噪，最终为WikiData中25类共1.278×10⁵个实体收集1.8×10⁶幅图像。实验结果表明，与IMGpedia、VisualSem、Richpedia和MMKG这4种多模态知识图谱相比，利用该方法构建所得的知识图谱中实体对应的图像在准确性和多样性上更具优势，在下游任务链接预测中，通过引入该方法收集到的图像可使模型的预测链接准确性得到显著提升，在Hits@10的指标上取得59.74%的准确率，较对比方法提高12.7个百分点以上。

关键词: 多模态知识图谱, 符号接地, 模式迁移, 链接预测, 实体配图

Abstract: The core of constructing multi-modality knowledge graph is to ensure the correct and appropriate images match the entities in the knowledge graph.Existing entity image collection methods mainly use encyclopedias and image search engines as the source of images to serve as entity candidates;however, their application of image data elements is relatively simple in that they cannot accurately grasp the characteristics of image data sources, and their scalability is poor.Here, an entity image collection method based on multi-modality pattern transfer is proposed.The method extracts the corresponding semantic template from different types of head entities and transfers the visual mode to the image acquisition process of similar non-head entities.Semantic templates are used to build search engine search keywords, and visual modes are used to denoise the search results.Ultimately, the method collects 1.8×10⁶ images for 1.278×10⁵ entities in 25 categories of WikiData.The experimental results show that, compared with IMGpedia, VisualSem, Richpedia, and MMKG, the images corresponding to entities in the multi-modality knowledge graph constructed by the proposed method are more accurate with greater diversity.The accuracy of the link prediction in downstream task can be significantly improved by introducing the images collected by this method.In Hits@10, the accuracy of the index is 59.74%, which is at least 12.7 percentage points higher than that of the methods used for comparison.

Key words: multi-modality knowledge graph, symbol grounding, pattern transfer, link prediction, entity image collection

中图分类号:

TP391.1

蒋雪瑶, 力维辰, 刘井平, 李直旭, 肖仰华. 基于多模态模式迁移的知识图谱实体配图[J]. 计算机工程, 2022, 48(8): 70-76.

JIANG Xueyao, LI Weichen, LIU Jingping, LI Zhixu, XIAO Yanghua. Entity Image Collection Based on Multi-Modality Pattern Transfer[J]. Computer Engineering, 2022, 48(8): 70-76.

http://www.ecice06.com/CN/Y2022/V48/I8/70

图/表 10

20220825091137

20220825091141

20220825091145

20220825091148

20220825091152

20220825091155

20220825091159

20220825091202

20220825091206

20220825091210

参考文献

[1] 刘昱然.多模态领域知识图谱构建方法及应用研究[D].银川:北方民族大学, 2020. LIU Y R.Research on the construction method and application of multi-modality domain knowledge graph[D].Yingchuan:Northern University for nationalities, 2020.(in Chinese)
[2] 何俊, 张彩庆, 李小珍, 等.面向深度学习的多模态融合技术研究综述[J].计算机工程, 2020, 46(5):1-11. HE J, ZHANG C Q, LI X Z, et al.Survey of research on multimodal fusion technology for deep learning[J].Computer Engineering, 2020, 46(5):1-11.(in Chinese)
[3] BALTRUSAITIS T, AHUJA C, MORENCY L P.Multimodal machine learning:a survey and taxonomy[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(2):423-443.
[4] 卢超.基于多模态知识图谱的图像描述[D].石家庄:河北科技大学, 2020. LU C.Image description based on multimodal knowledge graph[D].Shijiazhuang:Hebei University of Science and Technology, 2020.(in Chinese)
[5] MOUSSELLY S H, BOTSCHEN T, GUREVYCH I, et al.A multimodal translation-based approach for knowledge graph representation learning[C]//Proceedings of the 17th Joint Conference on Lexical and Computational Semantics.Stroudsburg, USA:Association for Computational Linguistics, 2018:225-234.
[6] ZHU X R, LI Z X, WANG X D, et al.Multi-modal knowledge graph construction and application:a survey[EB/OL].[2022-01-10].https://arxiv.org/abs/2202.05786.
[7] 霍书全.人工智能符号接地问题研究的意义和挑战[J].上海师范大学学报(哲学社会科学版), 2019, 48(3):98-107. HUO S Q.The research significance and challenge of the AI symbol grounding problem[J].Journal of Shanghai Normal University (Philosophy &Social Sciences Edition), 2019, 48(3):98-107.(in Chinese)
[8] Wikipedia[M].[S.l.]:PediaPress, 2004.
[9] COMMONS W.Wikimedia[EB/OL].[2022-01-10].https://commons.wikimedia.org.
[10] FERRADA S, BUSTOS B, HOGAN A.IMGpedia:a linked dataset with content-based analysis of Wikimedia images[C]//Proceedings of the 21st International Symposium on Wearable Computers.Berlin, Germany:Springer, 2017:84-93.
[11] ALBERTS H, HUANG N Y, DESHPANDE Y, et al.VisualSem:a high-quality knowledge graph for vision and language[C]//Proceedings of the 1st Workshop on Multilingual Representation Learning.Stroudsburg, USA:Association for Computational Linguistics, 2021:1-15.
[12] WANG M, WANG H F, QI G L, et al.Richpedia:a large-scale, comprehensive multi-modal knowledge graph[J].Big Data Research, 2020, 22:1-10.
[13] GUNJAN V K, KUMARI M, KUMAR A, et al.Search engine optimization with Google[J].International Journal of Computer Science Issues, 2012, 9(1):206.
[14] OÑORO-RUBIO D, NIEPERT M, GARCÍA-DURÁN A, et al.Answering visual-relational queries in Web-extracted knowledge graphs[EB/OL].[2022-01-10].https://arxiv.org/abs/1709.02314.
[15] LIU Y, LI H, GARCIA-DURAN A, et al.MMKG:multi-modal knowledge graphs[C]//Proceedings of European Semantic Web Conference.Berlin, Germany:Springer, 2019:459-474.
[16] TORRALBA A, FERGUS R, FREEMAN W T.80 million tiny images:a large data set for nonparametric object and scene recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(11):1958-1970.
[17] CHEN X L, SHRIVASTAVA A, GUPTA A.NEIL:extracting visual knowledge from Web data[C]//Proceedings of 2013 IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2013:1409-1416.
[18] RADFORD A, KIM J W, HALLACY C, et al.Learning transferable visual models from natural language supervision[EB/OL].[2022-01-10].https://arxiv.org/abs/2103.00020.
[19] SIMONYAN K, ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].[2022-01-10].https://arxiv.org/abs/1409.1556.
[20] BREUNIG M M, KRIEGEL H P, NG R T, et al.LOF:identifying density-based local outliers[C]//Proceedings of 2000 ACM SIGMOD International Conference on Management of Data.New York, USA:ACM Press, 2000:93-104.
[21] ZHAO Z H, GUO S Q, XU Q L, et al.G-Means:a clustering algorithm for intrusion detection[M]//KOPPEN M, KASABOV N, COGHILL G.Advances in neuro-information processing.Berlin, Germany:Springer, 2009:563-570.
[22] UYAR A, KARAPINAR R.Investigating the precision of Web image search engines for popular and less popular entities[J].Journal of Information Science, 2017, 43(3):378-392.
[23] VRANDEČIĆD.Wikidata:a new platform for collaborative data collection[C]//Proceedings of the 21st International Conference on World Wide Web.New York, USA:ACM Press, 2012:1063-1064.
[24] JEONG J W, WANG X J, LEE D H.Towards measuring the visualness of a concept[C]//Proceedings of the 21st ACM International Conference on Information and Knowledge Management.New York, USA:ACM Press, 2012:2415-2418.
[25] LIU P F, YUAN W Z, FU J L, et al.Pre-train, prompt, and predict:a systematic survey of prompting methods in natural language processing[EB/OL].[2022-01-10].https://arxiv.org/abs/2107.13586.
[26] BOLLACKER K, EVANS C, PARITOSH P, et al.Freebase:a collaboratively created graph database for structuring human knowledge[C]//Proceedings of 2008 ACM SIGMOD International Conference on Management of Data.New York, USA:ACM Press, 2008:1247-1250.
[27] TSIMPOUKELLI M, MENICK J, CABI S, et al.Multimodal few-shot learning with Frozen language models[EB/OL].[2022-01-10].https://arxiv.org/abs/2106.13884.
[28] DEVLIN J, CHANG M W, LEE K, et al.BERT:pre-training of deep bidirectional transformers for language understanding[EB/OL].[2022-01-10].https://arxiv.org/abs/1810.04805.

选择文件类型/文献管理软件名称

选择包含的内容

基于多模态模式迁移的知识图谱实体配图

Entity Image Collection Based on Multi-Modality Pattern Transfer

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献

相关文章 5

编辑推荐

Metrics

本文评价

[1]	潘乐, 李弼程, 万旺, 曾荣燊. 基于关系强度理论与反馈机制的信息传播动态网络表示[J]. 计算机工程, 2023, 49(2): 246-253.
[2]	陈恒, 王思懿, 李冠宇, 祁瑞华, 杨晨, 王维美. 基于四元数胶囊网络的知识图谱补全模型[J]. 计算机工程, 2022, 48(2): 40-46,64.
[3]	王维美, 史一民, 李冠宇. 改进的胶囊网络知识图谱补全方法[J]. 计算机工程, 2020, 46(8): 21-26.
[4]	伍杰华, 熊云艳, 张顶, 陈嘉志. 融合多元影响力节点识别指标MPR的链接预测[J]. 计算机工程, 2020, 46(4): 301-308,315.
[5]	蔡小雨,陈可佳,安琛. 采用群体信息的二部图链接预测方法[J]. 计算机工程, 2016, 42(10): 187-191.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于多模态模式迁移的知识图谱实体配图

Entity Image Collection Based on Multi-Modality Pattern Transfer

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献

相关文章 5

编辑推荐

Metrics

本文评价