Knowledge Graph Completion Based on Contrastive Learning and Language Model-Enhanced Embedding

doi:10.19678/j.issn.1000-3428.0067543

Abstract

Abstract:

A knowledge graph is a structured knowledge base comprising various types of knowledge or data units obtained through extraction and other processes. It is used to describe and represent information, such as entities, concepts, facts, and relationships. The limitations of Natural Language Processing(NLP) technology and the presence of noise in the texts of various knowledge or information units affect the accuracy of information extraction. Existing Knowledge Graph Completion(KGC) methods typically account for only single structural information or text semantic information, whereas the structural and text semantic information in the entire knowledge graph is disregarded. Hence, a KGC model based on contrastive learning and language model-enhanced embedding is proposed. The input entities and relationships are obtained using a pretrained language model to obtain the textual semantic information of the entities and relationships. The distance scoring function of the translation model is used to capture the structured information in the knowledge graph. Two negative sampling methods for contrastive learning are used to fuse contrastive learning to train the model to improve its ability to represent positive and negative samples. Experimental results show that compared with the Bidirectional Encoder Representations from Transformers for Knowledge Graph completion(KG-BERT) model, this model improves the average proportion of triple with ranking less than or equal to 10(Hits@10) indicator by 31% and 23% on the WN18RR and FB15K-237 datasets, respectively, thus demonstrating its superiority over other similar models.

Key words: Knowledge Graph Completion(KGC), knowledge graph, contrastive learning, pretrained language model, link prediction

摘要：

知识图谱是由各种知识或数据单元经过抽取等处理而组成的一种结构化知识库, 用于描述和表示实体、概念、事实和关系等信息。自然语言处理技术的限制和各种知识或信息单元文本本身的噪声都会使信息抽取的准确性受到一定程度的影响。现有的知识图谱补全方法通常只考虑单一结构信息或者文本语义信息, 忽略了整个知识图谱中同时存在的结构信息与文本语义信息。针对此问题, 提出一种基于语言模型增强嵌入与对比学习的知识图谱补全(KGC)模型。将输入的实体和关系通过预训练语言模型获取实体和关系的文本语义信息, 利用翻译模型的距离打分函数捕获知识图谱中的结构信息, 使用2种用于对比学习的负采样方法融合对比学习来训练模型以提高模型对正负样本的表征能力。实验结果表明, 与基于来自Transformer的双向编码器表示的知识图谱补全(KG-BERT)模型相比, 在WN18RR和FB15K-237数据集上该模型链接预测的排名小于等于10的三元组的平均占比(Hits@10)分别提升了31%和23%, 明显优于对比模型。

关键词: 知识图谱补全, 知识图谱, 对比学习, 预训练语言模型, 链接预测

Hongchen ZHANG, Linyu LI, Li YANG, Chenjun SAN, Chunlin YIN, Bing YAN, Hong YU, Xuan ZHANG. Knowledge Graph Completion Based on Contrastive Learning and Language Model-Enhanced Embedding[J]. Computer Engineering, 2024, 50(4): 168-176.

张洪程, 李林育, 杨莉, 伞晨峻, 尹春林, 颜冰, 于虹, 张璇. 基于对比学习与语言模型增强嵌入的知识图谱补全[J]. 计算机工程, 2024, 50(4): 168-176.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0067543

http://www.ecice06.com/EN/Y2024/V50/I4/168

Figures/Tables 10

References 29

1	JI S X, PAN S R, CAMBRIA E, et al. A survey on knowledge graphs: representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33 (2): 494- 514. doi: 10.1109/TNNLS.2021.3070843
2	王昊奋, 丁军, 胡芳槐, 等. 大规模企业级知识图谱实践综述. 计算机工程, 2020, 46 (7): 1- 13. URL
	WANG H F, DING J, HU F H, et al. Survey on large scale enterprise-level knowledge graph practices. Computer Engineering, 2020, 46 (7): 1- 13. URL
3	陶天一, 王清钦, 付聿炜, 等. 基于知识图谱的金融新闻个性化推荐算法. 计算机工程, 2021, 47 (6): 98-103, 114. URL
	TAO T Y, WANG Q Q, FU Y W, et al. Personalized recommendation algorithm for financial news based on knowledge graph. Computer Engineering, 2021, 47 (6): 98-103, 114. URL
4	YANG Y H, HUANG C, XIA L H, et al. Knowledge graph contrastive learning for recommendation[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM Press, 2022: 1434-1443.
5	吴天波, 周欣, 程军军, 等. 基于位置和注意力联合表示的知识图谱问答. 计算机工程, 2022, 48 (8): 98-104, 112. URL
	WU T B, ZHOU X, CHENG J J, et al. Knowledge graph question-answering based on joint location and attention representation. Computer Engineering, 2022, 48 (8): 98-104, 112. URL
6	SHEN T, ZHANG F, CHENG J W. A comprehensive overview of knowledge graph completion. Knowledge-Based Systems, 2022, 255, 109597. doi: 10.1016/j.knosys.2022.109597
7	BORDES A, USUNIER N, GARCIA-DURAN A, et al. Translating embeddings for modeling multi-relational data[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2013: 2787-2795.
8	YAO L, MAO C, LUO Y. KG-BERT: BERT for knowledge graph completion[EB/OL]. [2023-04-19]. https://arxiv.org/abs/1909.03193.
9	MIKOLOV T, SUTSKEVER I, CHEN K, et al. Distributed representations of words and phrases and their compositionality[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2013: 3111-3119.
10	WANG Z, ZHANG J W, FENG J L, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the 28th AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2014: 1112-1119.
11	LIN Y K, LIU Z Y, SUN M S, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the 29th AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2015: 2181-2187.
12	JI G L, HE S Z, XU L H, et al. Knowledge graph embedding via dynamic mapping matrix[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Stroudsburg, USA: Association for Computational Linguistics, 2015: 687-696.
13	孟小艳, 蒋同海, 周喜, 等. 一种改进的自适应知识图谱嵌入式表示方法. 计算机应用研究, 2021, 38 (1): 39- 43. URL
	MENG X Y, JIANG T H, ZHOU X, et al. Improved adaptive embedding method for knowledge graph representation. Application Research of Computers, 2021, 38 (1): 39- 43. URL
14	YANG B S, YIH W T, HE X D, et al. Embedding entities and relations for learning and inference in knowledge bases[EB/OL]. [2023-04-11]. https://arxiv.org/abs/1412.6575.
15	TROUILLON T, WELBL J, RIEDEL S, et al. Complex embeddings for simple link prediction[C]//Proceedings of the 33rd International Conference on Machine Learning. New York, USA: ACM Press, 2016: 2071-2080.
16	BALAŽEVIĆ I, ALLEN C, HOSPEDALES T M. TuckER: tensor factorization for knowledge graph completion[C]//Proceedings of 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Philadelphia, USA: Association for Computational Linguistics, 2019: 5185-5194.
17	赵博, 王宇嘉, 倪骥. 知识图谱的增强CP分解链接预测方法. 计算机应用研究, 2023, 40 (5): 1396- 1401. URL
	ZHAO B, WANG Y J, NI J. Enhanced CP decomposition link prediction method for knowledge graph. Computer Application Research, 2023, 40 (5): 1396- 1401. URL
18	DETTMERS T, MINERVINI P, STENETORP P, et al. Convolutional 2D knowledge graph embeddings[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence and 13th Innovative Applications of Artificial Intelligence Conference and 8th AAAI Symposium on Educational Advances in Artificial Intelligence. Palo Alto, USA: AAAI Press, 2018: 1811-1818.
19	VASHISHTH S, SANYAL S, NITIN V, et al. Composition-based multi-relational graph convolutional networks[EB/OL]. [2023-04-19]. https://arxiv.org/abs/1911.03082.
20	NGUYEN D Q, NGUYEN T D, NGUYEN D Q, et al. A novel embedding model for knowledge base completion based on convolutional neural network[EB/OL]. [2023-04-19]. https://arxiv.org/abs/1712.02121.
21	SOCHER R, CHEN D Q, MANNING C D, et al. Reasoning with neural tensor networks for knowledge base completion[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2013: 926-934
22	ZHANG Z Q, WANG J, YE J P, et al. Rethinking graph convolutional networks in knowledge graph completion[C]//Proceedings of ACM Web Conference 2022. New York, USA: ACM Press, 2022: 798-807.
23	邹长龙, 安敬民, 李冠宇. 基于邻域聚合与CNN的知识图谱实体类型补全. 计算机工程, 2023, 49 (3): 134- 141. URL
	ZOU C L, AN J M, LI G Y. Knowledge graph entity type completion based on neighborhood aggregation and CNN. Computer Engineering, 2023, 49 (3): 134- 141. URL
24	ZHANG Y, YAO Q, SHAO Y, et al. NSCaching: simple and efficient negative sampling for knowledge graph embedding[C]//Proceedings of the 35th International Conference on Data Engineering. Washington D. C., USA: IEEE Press, 2019: 614-625.
25	KIM B, HONG T, KO Y, et al. Multi-task learning for knowledge graph completion with pre-trained language models[C]//Proceedings of the 28th International Conference on Computational Linguistics. Stroudsburg, USA: International Committee on Computational Linguistics, 2020: 1737-1743.
26	LI D, YI M, HE Y Q. LP-BERT: multi-task pre-training knowledge graph BERT for link prediction[EB/OL]. [2023-04-19]. https://arxiv.org/abs/2201.04843v1.
27	ZKE A, GROSS S, MASSA F, et al. PyTorch: an imperative style, high performance deep learning library[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2019, 32: 18-30.
28	TOUTANOVA K, CHEN D Q, PANTEL P, et al. Representing text for joint embedding of text and knowledge bases[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: Association for Computational Linguistics, 2015: 1499-1509.
29	WANG F, LIU H P. Understanding the behaviour of contrastive loss[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 2495-2504.

[1]	LI Zhengxue, LI Zhiming, PENG Dezhong, CHEN Jie. User Classification of Social Networks Based on Feature Contrastive Learning and Graph Convolution [J]. Computer Engineering, 2024, 50(4): 258-266.
[2]	Guanrong WU, Yuanxiang LI, Yilin WANG, Yuhan LU, Xiuhua CHEN. Few-Shot Metal Surface Defect Classification Based on Contrastive Learning [J]. Computer Engineering, 2024, 50(3): 36-43.
[3]	Faxin CAO, Yuanyuan SUN, Zhizheng WANG, Dinghao PAN, Hongfei LIN. Similar Case Matching Model for Lending Cases [J]. Computer Engineering, 2024, 50(1): 306-312.
[4]	Zhibao WANG, Shutao JIANG, Fei LI, Juntao GAO, Qiang MA, Bin YANG. Entity Alignment of Petroleum Data Assets Graph Based on Multi-Neighborhood Awareness [J]. Computer Engineering, 2024, 50(1): 339-347.
[5]	Sha ZHOU, Guowei SHEN, Chun GUO. Vulnerability Information Completion Based on Security Knowledge Graph and Reverse Features [J]. Computer Engineering, 2024, 50(1): 145-155.
[6]	Jiajing GU, Dan YANG, Tiezheng NIE, Yue KOU. Recommendation Algorithm Based on Multi-view Fusion Cross-layer Contrastive Learning [J]. Computer Engineering, 2024, 50(1): 120-128.
[7]	Hongpeng LI, Bo MA, Yating YANG, Lei WANG, Zhen WANG, Xiao LI. Document-level Event Extraction Method Based on Slot Semantic Enhanced Prompt Learning [J]. Computer Engineering, 2023, 49(9): 23-31.
[8]	Guohua SUI, Taoran LI, Hao LIU, Lin CHEN, Wei WANG. Research on Domain Knowledge Graph Inference Technology Based on Graph Representation Learning [J]. Computer Engineering, 2023, 49(9): 89-98.
[9]	Zhangjie RAN, Linfu SUN, Yisheng ZOU, Yulin MA. Few-Shot Knowledge Graph Completion Model Based on Relation Learning Network [J]. Computer Engineering, 2023, 49(9): 52-59.
[10]	Kun MA, Jingmin AN, Guanyu LI. Knowledge Graph Completion with Dynamically Aggregating Context of Entity and Relation [J]. Computer Engineering, 2023, 49(8): 77-84, 95.
[11]	DAI Zuhua, LIU Yuanyuan, DI Shilong. Semantic Enhanced Aspect-Level Text Sentiment Analysis of Graph Neural Networks [J]. Computer Engineering, 2023, 49(6): 71-80.
[12]	YE Qi, ZHANG Yiqian, RUAN Tong, DU Wen. Quality Verification Method for Knowledge Graph Based on Semantic and Structural Trustworthiness [J]. Computer Engineering, 2023, 49(5): 48-55.
[13]	TANG Yan, LU Manxu. Recommendation System Based on Knowledge Graph and Deep Ripple Network [J]. Computer Engineering, 2023, 49(5): 63-72,80.
[14]	LI Xiaoteng, ZHANG Panpan, GOU Zhinan, GAO Kai. Multi-Modal Named Entity Recognition Method Based on Multi-Task Learning [J]. Computer Engineering, 2023, 49(4): 114-119.
[15]	ZHAI Sheping, ZHANG Yuhang, BAI Xiaoxia. Knowledge Graph Embedding Negative Sampling Method Fused with Entity Neighborhood Information [J]. Computer Engineering, 2023, 49(3): 95-104.

Please choose a citation manager

Content to export