基于对比学习与语言模型增强嵌入的知识图谱补全

doi:10.19678/j.issn.1000-3428.0067543

计算机工程 ›› 2024, Vol. 50 ›› Issue (4): 168-176. doi: 10.19678/j.issn.1000-3428.0067543

基于对比学习与语言模型增强嵌入的知识图谱补全

张洪程¹, 李林育², 杨莉³, 伞晨峻¹, 尹春林³, 颜冰¹, 于虹³, 张璇^2,4,5

1. 云南电网有限责任公司政企部, 云南昆明 650032;
2. 云南大学软件学院, 云南昆明 650091;
3. 云南电网有限责任公司电力科学研究院, 云南昆明 650217;
4. 云南省软件工程重点实验室, 云南昆明 650091;
5. 跨境网络空间安全教育部工程研究中心, 云南昆明 650091

收稿日期:2023-05-04 修回日期:2023-07-04 发布日期:2024-04-15
通讯作者: 张璇,E-mail:zhxuan@ynu.edu.cn E-mail:zhxuan@ynu.edu.cn
基金资助:
国家自然科学基金(61862063,61502413,61262025);云南电网有限责任公司创新项目(YNKJXM20222254);云南省中青-学术和技术带头人后备人才项目(202205AC160040);云南省院士专家工作站项目(202205AF150006);云南省科技计划重大专项计划项目(202202AE090066);云南省教育厅科学研究基金(2023Y0256);云南大学软件学院"知识驱动智能软件工程科研创新团队"项目。

Knowledge Graph Completion Based on Contrastive Learning and Language Model-Enhanced Embedding

ZHANG Hongchen¹, LI Linyu², YANG Li³, SAN Chenjun¹, YIN Chunlin³, YAN Bing¹, YU Hong³, ZHANG Xuan^2,4,5

1. Policy Research and Enterprise Management Department, Yunnan Power Grid Co., Ltd., Kunming 650032, Yunnan, China;
2. School of Software, Yunnan University, Kunming 650091, Yunnan, China;
3. Electric Power Research Institute, Yunnan Power Grid Co., Ltd., Kunming 650217, Yunnan, China;
4. Key Laboratory of Software Engineering of Yunnan Province, Kunming 650091, Yunnan, China;
5. Engineering Research Center of Cyberspace, Kunming 650091, Yunnan, China

Received:2023-05-04 Revised:2023-07-04 Published:2024-04-15

摘要/Abstract

摘要： 知识图谱是由各种知识或数据单元经过抽取等处理而组成的一种结构化知识库,用于描述和表示实体、概念、事实和关系等信息。自然语言处理技术的限制和各种知识或信息单元文本本身的噪声都会使信息抽取的准确性受到一定程度的影响。现有的知识图谱补全方法通常只考虑单一结构信息或者文本语义信息,忽略了整个知识图谱中同时存在的结构信息与文本语义信息。针对此问题,提出一种基于语言模型增强嵌入与对比学习的知识图谱补全(KGC)模型。将输入的实体和关系通过预训练语言模型获取实体和关系的文本语义信息,利用翻译模型的距离打分函数捕获知识图谱中的结构信息,使用2种用于对比学习的负采样方法融合对比学习来训练模型以提高模型对正负样本的表征能力。实验结果表明,与基于来自Transformer的双向编码器表示的知识图谱补全(KG-BERT)模型相比,在WN18RR和FB15K-237数据集上该模型链接预测的排名小于等于10的三元组的平均占比(Hits@10)分别提升了31%和23%,明显优于对比模型。

关键词: 知识图谱补全, 知识图谱, 对比学习, 预训练语言模型, 链接预测

Abstract: A knowledge graph is a structured knowledge base comprising various types of knowledge or data units obtained through extraction and other processes. It is used to describe and represent information, such as entities, concepts, facts, and relationships. The limitations of Natural Language Processing(NLP) technology and the presence of noise in the texts of various knowledge or information units affect the accuracy of information extraction. Existing Knowledge Graph Completion(KGC) methods typically account for only single structural information or text semantic information, whereas the structural and text semantic information in the entire knowledge graph is disregarded. Hence, a KGC model based on contrastive learning and language model-enhanced embedding is proposed. The input entities and relationships are obtained using a pretrained language model to obtain the textual semantic information of the entities and relationships. The distance scoring function of the translation model is used to capture the structured information in the knowledge graph. Two negative sampling methods for contrastive learning are used to fuse contrastive learning to train the model to improve its ability to represent positive and negative samples. Experimental results show that compared with the Bidirectional Encoder Representations from Transformers for Knowledge Graph completion(KG-BERT) model, this model improves the average proportion of triple with ranking less than or equal to 10(Hits@10) indicator by 31% and 23% on the WN18RR and FB15K-237 datasets, respectively, thus demonstrating its superiority over other similar models.

Key words: Knowledge Graph Completion(KGC), knowledge graph, contrastive learning, pretrained language model, link prediction

中图分类号:

TP399

张洪程, 李林育, 杨莉, 伞晨峻, 尹春林, 颜冰, 于虹, 张璇. 基于对比学习与语言模型增强嵌入的知识图谱补全[J]. 计算机工程, 2024, 50(4): 168-176.

ZHANG Hongchen, LI Linyu, YANG Li, SAN Chenjun, YIN Chunlin, YAN Bing, YU Hong, ZHANG Xuan. Knowledge Graph Completion Based on Contrastive Learning and Language Model-Enhanced Embedding[J]. Computer Engineering, 2024, 50(4): 168-176.

https://www.ecice06.com/CN/Y2024/V50/I4/168

参考文献

[1] JI S X, PAN S R, CAMBRIA E, et al. A survey on knowledge graphs:representation, acquisition, and applications[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(2):494-514.
[2] 王昊奋, 丁军, 胡芳槐, 等. 大规模企业级知识图谱实践综述[J]. 计算机工程, 2020, 46(7):1-13. WANG H F, DING J, HU F H, et al. Survey on large scale enterprise-level knowledge graph practices[J]. Computer Engineering, 2020, 46(7):1-13.(in Chinese)
[3] 陶天一, 王清钦, 付聿炜, 等. 基于知识图谱的金融新闻个性化推荐算法[J]. 计算机工程, 2021, 47(6):98-103, 114. TAO T Y, WANG Q Q, FU Y W, et al. Personalized recommendation algorithm for financial news based on knowledge graph[J]. Computer Engineering, 2021, 47(6):98-103, 114.(in Chinese)
[4] YANG Y H, HUANG C, XIA L H, et al. Knowledge graph contrastive learning for recommendation[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA:ACM Press, 2022:1434-1443.
[5] 吴天波, 周欣, 程军军, 等. 基于位置和注意力联合表示的知识图谱问答[J]. 计算机工程, 2022, 48(8):98-104, 112. WU T B, ZHOU X, CHENG J J, et al. Knowledge graph question-answering based on joint location and attention representation[J]. Computer Engineering, 2022, 48(8):98-104, 112.(in Chinese)
[6] SHEN T, ZHANG F, CHENG J W. A comprehensive overview of knowledge graph completion[J]. Knowledge-Based Systems, 2022, 255:109597.
[7] BORDES A, USUNIER N, GARCIA-DURAN A, et al. Translating embeddings for modeling multi-relational data[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. New York, USA:ACM Press, 2013:2787-2795.
[8] YAO L, MAO C, LUO Y. KG-BERT:BERT for knowledge graph completion[EB/OL].[2023-04-19]. https://arxiv.org/abs/1909.03193.
[9] MIKOLOV T, SUTSKEVER I, CHEN K, et al. Distributed representations of words and phrases and their compositionality[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. New York, USA:ACM Press, 2013:3111-3119.
[10] WANG Z, ZHANG J W, FENG J L, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the 28th AAAI Conference on Artificial Intelligence. Palo Alto, USA:AAAI Press, 2014:1112-1119.
[11] LIN Y K, LIU Z Y, SUN M S, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the 29th AAAI Conference on Artificial Intelligence. Palo Alto, USA:AAAI Press, 2015:2181-2187.
[12] JI G L, HE S Z, XU L H, et al. Knowledge graph embedding via dynamic mapping matrix[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Stroudsburg, USA:Association for Computational Linguistics, 2015:687-696.
[13] 孟小艳, 蒋同海, 周喜, 等. 一种改进的自适应知识图谱嵌入式表示方法[J]. 计算机应用研究, 2021, 38(1):39-43. MENG X Y, JIANG T H, ZHOU X, et al. Improved adaptive embedding method for knowledge graph representation[J]. Application Research of Computers, 2021, 38(1):39-43.(in Chinese)
[14] YANG B S, YIH W T, HE X D, et al. Embedding entities and relations for learning and inference in knowledge bases[EB/OL].[2023-04-11]. https://arxiv.org/abs/1412.6575.
[15] TROUILLON T, WELBL J, RIEDEL S, et al. Complex embeddings for simple link prediction[C]//Proceedings of the 33rd International Conference on Machine Learning. New York, USA:ACM Press, 2016:2071-2080.
[16] BALAŽEVIĆ I, ALLEN C, HOSPEDALES T M. TuckER:tensor factorization for knowledge graph completion[C]//Proceedings of 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Philadelphia, USA:Association for Computational Linguistics, 2019:5185-5194.
[17] 赵博,王宇嘉,倪骥.知识图谱的增强CP分解链接预测方法[J]. 计算机应用研究, 2023, 40(5):1396-1401. ZHAO B, WANG Y J, NI J. Enhanced CP decomposition link prediction method for knowledge graph[J]. Computer Application Research, 2023, 40(5):1396-1401.(in Chinese)
[18] DETTMERS T, MINERVINI P, STENETORP P, et al. Convolutional 2D knowledge graph embeddings[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence and 13th Innovative Applications of Artificial Intelligence Conference and 8th AAAI Symposium on Educational Advances in Artificial Intelligence. Palo Alto, USA:AAAI Press, 2018:1811-1818.
[19] VASHISHTH S, SANYAL S, NITIN V, et al. Composition-based multi-relational graph convolutional networks[EB/OL].[2023-04-19]. https://arxiv.org/abs/1911.03082.
[20] NGUYEN D Q, NGUYEN T D, NGUYEN D Q, et al. A novel embedding model for knowledge base completion based on convolutional neural network[EB/OL].[2023-04-19]. https://arxiv.org/abs/1712.02121.
[21] SOCHER R, CHEN D Q, MANNING C D, et al. Reasoning with neural tensor networks for knowledge base completion[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. New York, USA:ACM Press, 2013:926-934
[22] ZHANG Z Q, WANG J, YE J P, et al. Rethinking graph convolutional networks in knowledge graph completion[C]//Proceedings of ACM Web Conference 2022. New York, USA:ACM Press, 2022:798-807.
[23] 邹长龙, 安敬民, 李冠宇. 基于邻域聚合与CNN的知识图谱实体类型补全[J]. 计算机工程, 2023, 49(3):134-141. ZOU C L, AN J M, LI G Y. Knowledge graph entity type completion based on neighborhood aggregation and CNN[J]. Computer Engineering, 2023, 49(3):134-141.(in Chinese)
[24] ZHANG Y, YAO Q, SHAO Y, et al. NSCaching:simple and efficient negative sampling for knowledge graph embedding[C]//Proceedings of the 35th International Conference on Data Engineering. Washington D.C., USA:IEEE Press, 2019:614-625.
[25] KIM B, HONG T, KO Y, et al. Multi-task learning for knowledge graph completion with pre-trained language models[C]//Proceedings of the 28th International Conference on Computational Linguistics. Stroudsburg, USA:International Committee on Computational Linguistics, 2020:1737-1743.
[26] LI D, YI M, HE Y Q. LP-BERT:multi-task pre-training knowledge graph BERT for link prediction[EB/OL].[2023-04-19]. https://arxiv.org/abs/2201.04843v1.
[27] ZKE A, GROSS S, MASSA F, et al. PyTorch:an imperative style, high performance deep learning library[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems. New York, USA:ACM Press, 2019, 32:18-30.
[28] TOUTANOVA K, CHEN D Q, PANTEL P, et al. Representing text for joint embedding of text and knowledge bases[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA:Association for Computational Linguistics, 2015:1499-1509.
[29] WANG F, LIU H P. Understanding the behaviour of contrastive loss[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA:IEEE Press, 2021:2495-2504.

选择文件类型/文献管理软件名称

选择包含的内容

基于对比学习与语言模型增强嵌入的知识图谱补全

Knowledge Graph Completion Based on Contrastive Learning and Language Model-Enhanced Embedding

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	李泽霖, 吕兆峰, 陈富强, 李克. 基于多跳信息融合的实体对齐模型[J]. 计算机工程, 2024, 50(9): 142-152.
[2]	汤志康, 武毓琦, 李春英, 汤庸. 基于知识图谱卷积网络的学习资源推荐[J]. 计算机工程, 2024, 50(9): 153-160.
[3]	李华昱, 张智康, 闫阳, 岳阳. 基于知识图谱增强的领域多模态实体识别[J]. 计算机工程, 2024, 50(8): 31-39.
[4]	高爽, 史轶伦, 徐巧枝, 于磊. 基于对比学习的非对称编解码结构的心脏MRI分割研究[J]. 计算机工程, 2024, 50(8): 290-300.
[5]	陈宇航, 杨勇, 先木斯亚·买买提明, 帕力旦·吐尔逊, 樊小超, 任鸽, 刁宇峰. 基于主题感知和语义增强的作文自动评分方法[J]. 计算机工程, 2024, 50(8): 363-371.
[6]	李惠乾, 钟柏昌. 教育知识图谱: 研究进展与未来发展——基于2013—2023年中文核心期刊载文的分析[J]. 计算机工程, 2024, 50(7): 1-12.
[7]	王炼红, 林飞鹏, 李潇瑶, 谌桂枝, 周莉. 融入课程知识图谱的KMAKT预测[J]. 计算机工程, 2024, 50(7): 23-31.
[8]	钱来, 赵卫伟. 基于对比学习和注意力机制的文本分类方法[J]. 计算机工程, 2024, 50(7): 104-111.
[9]	刘娟, 段友祥, 陆誉翕, 张鲁. 引入知识增强和对比学习的知识图谱补全[J]. 计算机工程, 2024, 50(7): 112-122.
[10]	陈佳玉, 王元龙, 张虎. 基于文本知识增强的问题生成模型[J]. 计算机工程, 2024, 50(6): 86-93.
[11]	武星, 殷浩宇, 姚骏峰, 李卫民, 钱权. 面向视频数据的多模态情感分析[J]. 计算机工程, 2024, 50(6): 218-227.
[12]	江惠珍, 孙艳春, 黄罡. 基于知识图谱的GitHub层次化学习和检索服务[J]. 计算机工程, 2024, 50(5): 16-25.
[13]	张宝鑫, 杨丹, 聂铁铮, 寇月. 基于自监督的多视角图协同过滤推荐方法[J]. 计算机工程, 2024, 50(5): 100-110.
[14]	李田芳, 普园媛, 赵征鹏, 徐丹, 钱文华. 基于CLIP和双空间自适应归一化的图像翻译[J]. 计算机工程, 2024, 50(5): 229-240.
[15]	隗昊, 刁宏悦, 孔亮宸, 邓耀臣. 东北亚舆情文本细粒度命名实体识别方法研究[J]. 计算机工程, 2024, 50(5): 354-362.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于对比学习与语言模型增强嵌入的知识图谱补全

Knowledge Graph Completion Based on Contrastive Learning and Language Model-Enhanced Embedding

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价