Knowledge Graph Completion with Knowledge Enhancement and Contrastive Learning

doi:10.19678/j.issn.1000-3428.0068020

Abstract

Abstract:

Knowledge Graph Completion(KGC) is an important means of improving the quality of KGs. Existing methods for KGC are mainly divided into structure- and description-based methods. The structure-based methods perform poorly in the inference of long-tailed entities commonly found in KGs, and the description-based methods are insufficient for descriptive information utilization and negative sample information learning. To address these challenges, this paper proposes a KGC method with knowledge enhancement and contrastive learning, named KEKGC. A specific template is designed to convert triples and their descriptive information into coherent natural language statements. These statements serve as input to a Pre-trained Language Model(PLM) through a manually defined template, enhancing the language model's comprehension of the structural and descriptive knowledge of each triple. On this basis, a contrastive learning framework is incorporated to improve the efficiency and accuracy of the link prediction task. This is done by building a memory bank to store entity embedding vectors, from which positive and negative samples are selected and trained with InfoNCE loss. Experimental results on WN18RR dataset show that compared with MEM-KGC, in the link prediction task, KEKGC improves Mean Reciprocal Rank(MRR) by 5.5, improves Hits@1, Hits@3, and Hits@10 metrics by 2.8, 0.7, and 4.2 percentage points, respectively, and the accuracy of the triple classification task reaches 94.1%. Hence, this method can achieve higher prediction accuracy and better generalization ability, especially for long-tailed entities, which can effectively improve graph completion.

Key words: Knowledge Graph(KG), Pre-trained Language Model(PLM), link prediction, contrastive learning, entity description

摘要：

知识图谱补全是提高知识图谱质量的重要手段, 主要分为基于结构和基于描述的方法。基于结构的补全方法对图谱中常见的长尾实体推理性能表现不佳, 基于描述的补全方法在描述信息利用和负样本信息学习方面存在不足。针对上述问题, 提出基于知识增强的知识图谱补全方法KEKGC。设计一种特定模板, 将三元组及其描述信息通过人工定义的模板转换为连贯的自然语言描述语句输入预训练语言模型, 增强语言模型对三元组结构知识与描述知识的理解能力。在此基础上, 提出一种对比学习框架来提高链接预测任务的效率与准确率, 通过建立记忆库存储实体嵌入向量, 从中选择正负样本并结合InfoNCE损失进行训练。实验结果显示, 相较于MEM-KGC, KEKGC在WN18RR数据集上链接预测任务的平均倒数秩(MRR)提升了5.5, Hits@1、Hits@3、Hits@10指标分别提升了2.8、0.7、4.2个百分点, 三元组分类任务准确率达到94.1%, 表明所提方法具有更高的预测准确率与更好的泛化能力, 尤其对于长尾实体, 能够有效提升图谱补全的效果与效率。

关键词: 知识图谱, 预训练语言模型, 链接预测, 对比学习, 实体描述

Juan LIU, Youxiang DUAN, Yuxi LU, Lu ZHANG. Knowledge Graph Completion with Knowledge Enhancement and Contrastive Learning[J]. Computer Engineering, 2024, 50(7): 112-122.

刘娟, 段友祥, 陆誉翕, 张鲁. 引入知识增强和对比学习的知识图谱补全[J]. 计算机工程, 2024, 50(7): 112-122.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0068020

https://www.ecice06.com/EN/Y2024/V50/I7/112

Figures/Tables 8

References 40

1	HOGAN A, BLOMQVIST E, COCHEZ M, et al. Knowledge graphs. ACM Computing Surveys, 54(4): 71.
2	王昊奋, 丁军, 胡芳槐, 等. 大规模企业级知识图谱实践综述. 计算机工程, 2020, 46(7): 1- 13. URL
	WANG H F, DING J, HU F H, et al. Survey on large scale enterprise-level knowledge graph practices. Computer Engineering, 2020, 46(7): 1- 13. URL
3	王萌, 王昊奋, 李博涵, 等. 新一代知识图谱关键技术综述. 计算机研究与发展, 2022, 59(9): 1947- 1965. URL
	WANG M, WANG H F, LI B H, et al. Survey on key technologies of new generation knowledge graph. Journal of Computer Research and Development, 2022, 59(9): 1947- 1965. URL
4	张吉祥, 张祥森, 武长旭, 等. 知识图谱构建技术综述. 计算机工程, 2022, 48(3): 23- 37. URL
	ZHANG J X, ZHANG X S, WU C X, et al. Survey of knowledge graph construction techniques. Computer Engineering, 2022, 48(3): 23- 37. URL
5	BOLLACKER K D, EVANS C, PARITOSH P, et al. Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of the 22nd AAAI Conference on Management of Data. New York, USA: AAAI Press, 2007: 1962-1963.
6	VRANDEČIĆ D, KRÖTZSCH M. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 2014, 57(10): 78- 85. doi: 10.1145/2629489
7	SUCHANEK F M, KASNECI G, WEIKUM G. Yago: a core of semantic knowledge[C]//Proceedings of the 16th International Conference on World Wide Web. New York, USA: ACM Press, 2007: 697-706.
8	JI S X, PAN S R, CAMBRIA E, et al. A survey on knowledge graphs: representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(2): 494- 514. doi: 10.1109/TNNLS.2021.3070843
9	张金斗, 李京. 一种结合层次化类别信息的知识图谱表示学习方法. 软件学报, 2022, 33(9): 3331- 3346. URL
	ZHANG J D, LI J. Knowledge graph embedding combining with hierarchical type information. Journal of Software, 2022, 33(9): 3331- 3346. URL
10	BORDES A, USUNIER N, GARCIA-DURÁN A, et al. Translating embeddings for modeling multi-relational data[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2013: 2787-2795.
11	WANG Z, ZHANG J W, FENG J L, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the 28th AAAI Conference on Artificial Intelligence. New York, USA: ACM Press, 2014: 1112-1119.
12	LIN Y, LIU Z, SUN M, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New York, USA: AAAI Press, 2015: 1-7.
13	SUN Z Q, DENG Z H, NIE J Y, et al. RotatE: knowledge graph embedding by relational rotation in complex space[C]//Proceedings of the 7th International Conference on Learning Representations. New Orleans, USA: [s. n. ], 2019: 1-18.
14	CHAO L L, HE J S, WANG T F, et al. PairRE: knowledge graph embeddings via paired relation vectors[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Stroudsburg, USA: Association for Computational Linguistics, 2021: 4360-4369.
15	YU L, LUO Z, LIN D, et al. TripleRE: knowledge graph embeddings via triple relation vectors[EB/OL]. [2023-04-26]. https://arxiv. org/abs/2209.08271.
16	NICKEL M, TRESP V, KRIEGEL H P. A three-way model for collective learning on multi-relational data[C]//Proceedings of the 28th International Conference on Machine Learning. Madison, USA: Omni Press, 2011: 809-816.
17	YANG B S, YIH W, HE X D, et al. Embedding entities and relations for learning and inference in knowledge bases[EB/OL]. [2023-04-26]. https://arxiv.org/abs/1412.6575.
18	陈恒, 王思懿, 李冠宇, 等. 基于四元数胶囊网络的知识图谱补全模型. 计算机工程, 2022, 48(2): 40-46, 64. URL
	CHEN H, WANG S Y, LI G Y, et al. Knowledge graph completion model based on quaternion capsule network. Computer Engineering, 2022, 48(2): 40-46, 64. URL
19	XIE R, LIU Z, JIA J, et al. Representation learning of knowledge graphs with entity descriptions[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New York, USA: AAAI Press, 2016: 1-7.
20	RADFORD A, NARASIMHAN K, SALIMANS T, et al. Improving language understanding by generative pre-training[EB/OL]. [2023-04-26]. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf.
21	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding [C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, USA: Association for Computational Linguistics, 2019: 4171-4186.
22	PETERS M E, NEUMANN M, LYYER M, et al. Deep contextualized word representations[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, USA: Association for Computational Linguistics, 2018: 2227-2237.
23	LIU Y H, OTT M, GOYAL N, et al. RoBERTa: a robustly optimized BERT pretraining approach[EB/OL]. [2023-04-26]. https://arxiv.org/abs/1907.11692.
24	PETRONI F, ROCKTSCHEL T, LEWIS P, et al. Language models as knowledge bases?[EB/OL]. [2023-04-26]. https://arxiv.org/abs/1909.01066.
25	YAO L, MAO C, LUO Y. KG-BERT: BERT for knowledge graph completion[EB/OL]. [2023-04-26]. https://arxiv.org/abs/1909.03193.
26	KIM B, HONG T, KO Y, et al. Multi-task learning for knowledge graph completion with pre-trained language models[C]//Proceedings of the 28th International Conference on Computational Linguistics. Barcelona, Spain: [s. n. ], 2020: 1737-1743.
27	WANG B, SHEN T, LONG G, et al. Structure-augmented text representation learning for efficient knowledge graph completion[C]//Proceedings of the 2021 Web Conference. Washington D. C., USA: IEEE Press, 2021: 1737-1748.
28	WANG X Z, GAO T Y, ZHU Z C, et al. KEPLER: a unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 2021, 9, 176- 194. doi: 10.1162/tacl_a_00360
29	GUNEL B, DU J F, CONNEAU A, et al. Supervised contrastive learning for pre-trained language model fine-tuning[EB/OL]. [2023-04-26]. https://arxiv.org/abs/2011.01403
30	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the 38th International Conference on Machine Learning. Washington D. C., USA: IEEE Press, 2021: 8748-8763.
31	LIU S, FAN H, QIAN S, et al. Hit: hierarchical transformer with momentum contrast for video-text retrieval[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 11915-11925.
32	CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[C]//Proceedings of the 37th International Conference on Machine Learning. Washington D. C., USA: IEEE Press, 2020: 1597-1607.
33	GAO T, YAO X, CHEN D. SimCSE: simple contrastive learning of sentence embeddings[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. [S. l. ]: Association for Computational Linguistics, 2021: 6894-6910.
34	CHOI B, JANG D, KO Y. MEM-KGC: masked entity modeling for knowledge graph completion with pre-trained language model. IEEE Access, 2021, 9, 132025- 132032. doi: 10.1109/ACCESS.2021.3113329
35	NGUYEN T D, NGUYEN D Q, PHUNG D. A novel embedding model for knowledge base completion based on convolutional neural network[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, USA: Association for Computational Linguistics, 2018: 327-333.
36	BALAZEVIC I, ALLEN C, HOSPEDALES T. TuckER: tensor factorization for knowledge graph completion[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Washington D. C., USA: IEEE Press, 2019: 5185-5194.
37	LI L Y, ZHANG X, MA Y B, et al. A knowledge graph completion model based on contrastive learning and relation enhancement method. Knowledge-Based Systems, 2022, 256, 109889. doi: 10.1016/j.knosys.2022.109889
38	ADNAN Z, SUMMAYA S, JUNDE C, et al. Learning knowledge graph embeddings by deep relational roto-reflection. Knowledge-Based Systems, 2022, 252, 109451. doi: 10.1016/j.knosys.2022.109451
39	SOCHER R, CHEN D, MANNING C D, et al. Reasoning with neural tensor networks for knowledge base completion[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. Washington D. C., USA: IEEE Press, 2013: 926-934.
40	MILLER G A. WordNet: a lexical database for English. Communications of the ACM, 1995, 38(11): 39- 41. doi: 10.1145/219717.219748

[1]	TANG Zhikang, WU Yuqi, LI Chunying, TANG Yong. Recommendation of Learning Resource Based on Knowledge Graph Convolutional Network [J]. Computer Engineering, 2024, 50(9): 153-160.
[2]	Shuang GAO, Yilun SHI, Qiaozhi XU, Lei YU. Research on Cardiac MRI Segmentation Based on Asymmetric Encoding and Decoding Structure of Contrastive Learning [J]. Computer Engineering, 2024, 50(8): 290-300.
[3]	Huayu LI, Zhikang ZHANG, Yang YAN, Yang YUE. Enhanced Domain Multi-modal Entity Recognition Based on Knowledge Graph [J]. Computer Engineering, 2024, 50(8): 31-39.
[4]	Lai QIAN, Weiwei ZHAO. Text Classification Method Based on Contrastive Learning and Attention Mechanism [J]. Computer Engineering, 2024, 50(7): 104-111.
[5]	WU Xing, YIN Haoyu, YAO Junfeng, LI Weimin, QIAN Quan. Multimodal Sentiment Analysis for Video Data [J]. Computer Engineering, 2024, 50(6): 218-227.
[6]	CHEN Jiayu, WANG Yuanlong, ZHANG Hu. Question Generation Model Based on Text Knowledge Enhancement [J]. Computer Engineering, 2024, 50(6): 86-93.
[7]	LI Tianfang, PU Yuanyuan, ZHAO Zhengpeng, XU Dan, QIAN Wenhua. Image-to-Image Translation Based on CLIP and Dual-Spatially Adaptive Normalization [J]. Computer Engineering, 2024, 50(5): 229-240.
[8]	ZHANG Baoxin, YANG Dan, NIE Tiezheng, KOU Yue. Recommendation Method Based on Self-supervised Multi-view Graph Collaborative Filtering [J]. Computer Engineering, 2024, 50(5): 100-110.
[9]	LI Zhengxue, LI Zhiming, PENG Dezhong, CHEN Jie. User Classification of Social Networks Based on Feature Contrastive Learning and Graph Convolution [J]. Computer Engineering, 2024, 50(4): 258-266.
[10]	ZHANG Hongchen, LI Linyu, YANG Li, SAN Chenjun, YIN Chunlin, YAN Bing, YU Hong, ZHANG Xuan. Knowledge Graph Completion Based on Contrastive Learning and Language Model-Enhanced Embedding [J]. Computer Engineering, 2024, 50(4): 168-176.
[11]	Guanrong WU, Yuanxiang LI, Yilin WANG, Yuhan LU, Xiuhua CHEN. Few-Shot Metal Surface Defect Classification Based on Contrastive Learning [J]. Computer Engineering, 2024, 50(3): 36-43.
[12]	LIU Duo, ZHANG Dong, LI Guanyu. No-CapsE: A Quaternion Capsule Network Knowledge Graph Completion Model Based on Node Co-occurrence [J]. Computer Engineering, 2024, 50(10): 80-88.
[13]	WU Yongqing, WANG Yuhan, ZHU Yue. Click-Through Rate Estimation Model Based on User Multi-Type Feedback Behavior Sequences [J]. Computer Engineering, 2024, 50(10): 405-417.
[14]	Zhibao WANG, Shutao JIANG, Fei LI, Juntao GAO, Qiang MA, Bin YANG. Entity Alignment of Petroleum Data Assets Graph Based on Multi-Neighborhood Awareness [J]. Computer Engineering, 2024, 50(1): 339-347.
[15]	Faxin CAO, Yuanyuan SUN, Zhizheng WANG, Dinghao PAN, Hongfei LIN. Similar Case Matching Model for Lending Cases [J]. Computer Engineering, 2024, 50(1): 306-312.

Please choose a citation manager

Content to export