引入知识增强和对比学习的知识图谱补全

doi:10.19678/j.issn.1000-3428.0068020

摘要/Abstract

摘要：

知识图谱补全是提高知识图谱质量的重要手段, 主要分为基于结构和基于描述的方法。基于结构的补全方法对图谱中常见的长尾实体推理性能表现不佳, 基于描述的补全方法在描述信息利用和负样本信息学习方面存在不足。针对上述问题, 提出基于知识增强的知识图谱补全方法KEKGC。设计一种特定模板, 将三元组及其描述信息通过人工定义的模板转换为连贯的自然语言描述语句输入预训练语言模型, 增强语言模型对三元组结构知识与描述知识的理解能力。在此基础上, 提出一种对比学习框架来提高链接预测任务的效率与准确率, 通过建立记忆库存储实体嵌入向量, 从中选择正负样本并结合InfoNCE损失进行训练。实验结果显示, 相较于MEM-KGC, KEKGC在WN18RR数据集上链接预测任务的平均倒数秩(MRR)提升了5.5, Hits@1、Hits@3、Hits@10指标分别提升了2.8、0.7、4.2个百分点, 三元组分类任务准确率达到94.1%, 表明所提方法具有更高的预测准确率与更好的泛化能力, 尤其对于长尾实体, 能够有效提升图谱补全的效果与效率。

关键词: 知识图谱, 预训练语言模型, 链接预测, 对比学习, 实体描述

Abstract:

Knowledge Graph Completion(KGC) is an important means of improving the quality of KGs. Existing methods for KGC are mainly divided into structure- and description-based methods. The structure-based methods perform poorly in the inference of long-tailed entities commonly found in KGs, and the description-based methods are insufficient for descriptive information utilization and negative sample information learning. To address these challenges, this paper proposes a KGC method with knowledge enhancement and contrastive learning, named KEKGC. A specific template is designed to convert triples and their descriptive information into coherent natural language statements. These statements serve as input to a Pre-trained Language Model(PLM) through a manually defined template, enhancing the language model's comprehension of the structural and descriptive knowledge of each triple. On this basis, a contrastive learning framework is incorporated to improve the efficiency and accuracy of the link prediction task. This is done by building a memory bank to store entity embedding vectors, from which positive and negative samples are selected and trained with InfoNCE loss. Experimental results on WN18RR dataset show that compared with MEM-KGC, in the link prediction task, KEKGC improves Mean Reciprocal Rank(MRR) by 5.5, improves Hits@1, Hits@3, and Hits@10 metrics by 2.8, 0.7, and 4.2 percentage points, respectively, and the accuracy of the triple classification task reaches 94.1%. Hence, this method can achieve higher prediction accuracy and better generalization ability, especially for long-tailed entities, which can effectively improve graph completion.

Key words: Knowledge Graph(KG), Pre-trained Language Model(PLM), link prediction, contrastive learning, entity description

刘娟, 段友祥, 陆誉翕, 张鲁. 引入知识增强和对比学习的知识图谱补全[J]. 计算机工程, 2024, 50(7): 112-122.

Juan LIU, Youxiang DUAN, Yuxi LU, Lu ZHANG. Knowledge Graph Completion with Knowledge Enhancement and Contrastive Learning[J]. Computer Engineering, 2024, 50(7): 112-122.

https://www.ecice06.com/CN/Y2024/V50/I7/112

图/表 8

图1 KEKGC方法流程

Fig.1 Procedure of KEKGC method

图2 KEKGC采用对比学习预测尾实体的过程

Fig.2 The process of predicting tail entities using contrastive learning in KEKGC

图3 记忆库中存储向量的标签矩阵

Fig.3 Labeling matrix of storage vector in the memory bank

参考文献 40

1	HOGAN A, BLOMQVIST E, COCHEZ M, et al. Knowledge graphs. ACM Computing Surveys, 54(4): 71.
2	王昊奋, 丁军, 胡芳槐, 等. 大规模企业级知识图谱实践综述. 计算机工程, 2020, 46(7): 1- 13. URL
	WANG H F, DING J, HU F H, et al. Survey on large scale enterprise-level knowledge graph practices. Computer Engineering, 2020, 46(7): 1- 13. URL
3	王萌, 王昊奋, 李博涵, 等. 新一代知识图谱关键技术综述. 计算机研究与发展, 2022, 59(9): 1947- 1965. URL
	WANG M, WANG H F, LI B H, et al. Survey on key technologies of new generation knowledge graph. Journal of Computer Research and Development, 2022, 59(9): 1947- 1965. URL
4	张吉祥, 张祥森, 武长旭, 等. 知识图谱构建技术综述. 计算机工程, 2022, 48(3): 23- 37. URL
	ZHANG J X, ZHANG X S, WU C X, et al. Survey of knowledge graph construction techniques. Computer Engineering, 2022, 48(3): 23- 37. URL
5	BOLLACKER K D, EVANS C, PARITOSH P, et al. Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of the 22nd AAAI Conference on Management of Data. New York, USA: AAAI Press, 2007: 1962-1963.
6	VRANDEČIĆ D, KRÖTZSCH M. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 2014, 57(10): 78- 85. doi: 10.1145/2629489
7	SUCHANEK F M, KASNECI G, WEIKUM G. Yago: a core of semantic knowledge[C]//Proceedings of the 16th International Conference on World Wide Web. New York, USA: ACM Press, 2007: 697-706.
8	JI S X, PAN S R, CAMBRIA E, et al. A survey on knowledge graphs: representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(2): 494- 514. doi: 10.1109/TNNLS.2021.3070843
9	张金斗, 李京. 一种结合层次化类别信息的知识图谱表示学习方法. 软件学报, 2022, 33(9): 3331- 3346. URL
	ZHANG J D, LI J. Knowledge graph embedding combining with hierarchical type information. Journal of Software, 2022, 33(9): 3331- 3346. URL
10	BORDES A, USUNIER N, GARCIA-DURÁN A, et al. Translating embeddings for modeling multi-relational data[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2013: 2787-2795.
11	WANG Z, ZHANG J W, FENG J L, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the 28th AAAI Conference on Artificial Intelligence. New York, USA: ACM Press, 2014: 1112-1119.
12	LIN Y, LIU Z, SUN M, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New York, USA: AAAI Press, 2015: 1-7.
13	SUN Z Q, DENG Z H, NIE J Y, et al. RotatE: knowledge graph embedding by relational rotation in complex space[C]//Proceedings of the 7th International Conference on Learning Representations. New Orleans, USA: [s. n. ], 2019: 1-18.
14	CHAO L L, HE J S, WANG T F, et al. PairRE: knowledge graph embeddings via paired relation vectors[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Stroudsburg, USA: Association for Computational Linguistics, 2021: 4360-4369.
15	YU L, LUO Z, LIN D, et al. TripleRE: knowledge graph embeddings via triple relation vectors[EB/OL]. [2023-04-26]. https://arxiv. org/abs/2209.08271.
16	NICKEL M, TRESP V, KRIEGEL H P. A three-way model for collective learning on multi-relational data[C]//Proceedings of the 28th International Conference on Machine Learning. Madison, USA: Omni Press, 2011: 809-816.
17	YANG B S, YIH W, HE X D, et al. Embedding entities and relations for learning and inference in knowledge bases[EB/OL]. [2023-04-26]. https://arxiv.org/abs/1412.6575.
18	陈恒, 王思懿, 李冠宇, 等. 基于四元数胶囊网络的知识图谱补全模型. 计算机工程, 2022, 48(2): 40-46, 64. URL
	CHEN H, WANG S Y, LI G Y, et al. Knowledge graph completion model based on quaternion capsule network. Computer Engineering, 2022, 48(2): 40-46, 64. URL
19	XIE R, LIU Z, JIA J, et al. Representation learning of knowledge graphs with entity descriptions[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New York, USA: AAAI Press, 2016: 1-7.
20	RADFORD A, NARASIMHAN K, SALIMANS T, et al. Improving language understanding by generative pre-training[EB/OL]. [2023-04-26]. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf.
21	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding [C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, USA: Association for Computational Linguistics, 2019: 4171-4186.
22	PETERS M E, NEUMANN M, LYYER M, et al. Deep contextualized word representations[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, USA: Association for Computational Linguistics, 2018: 2227-2237.
23	LIU Y H, OTT M, GOYAL N, et al. RoBERTa: a robustly optimized BERT pretraining approach[EB/OL]. [2023-04-26]. https://arxiv.org/abs/1907.11692.
24	PETRONI F, ROCKTSCHEL T, LEWIS P, et al. Language models as knowledge bases?[EB/OL]. [2023-04-26]. https://arxiv.org/abs/1909.01066.
25	YAO L, MAO C, LUO Y. KG-BERT: BERT for knowledge graph completion[EB/OL]. [2023-04-26]. https://arxiv.org/abs/1909.03193.
26	KIM B, HONG T, KO Y, et al. Multi-task learning for knowledge graph completion with pre-trained language models[C]//Proceedings of the 28th International Conference on Computational Linguistics. Barcelona, Spain: [s. n. ], 2020: 1737-1743.
27	WANG B, SHEN T, LONG G, et al. Structure-augmented text representation learning for efficient knowledge graph completion[C]//Proceedings of the 2021 Web Conference. Washington D. C., USA: IEEE Press, 2021: 1737-1748.
28	WANG X Z, GAO T Y, ZHU Z C, et al. KEPLER: a unified model for knowledge embedding and pre-trained language representation. Transactions of the Association for Computational Linguistics, 2021, 9, 176- 194. doi: 10.1162/tacl_a_00360
29	GUNEL B, DU J F, CONNEAU A, et al. Supervised contrastive learning for pre-trained language model fine-tuning[EB/OL]. [2023-04-26]. https://arxiv.org/abs/2011.01403
30	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the 38th International Conference on Machine Learning. Washington D. C., USA: IEEE Press, 2021: 8748-8763.
31	LIU S, FAN H, QIAN S, et al. Hit: hierarchical transformer with momentum contrast for video-text retrieval[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 11915-11925.
32	CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[C]//Proceedings of the 37th International Conference on Machine Learning. Washington D. C., USA: IEEE Press, 2020: 1597-1607.
33	GAO T, YAO X, CHEN D. SimCSE: simple contrastive learning of sentence embeddings[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. [S. l. ]: Association for Computational Linguistics, 2021: 6894-6910.
34	CHOI B, JANG D, KO Y. MEM-KGC: masked entity modeling for knowledge graph completion with pre-trained language model. IEEE Access, 2021, 9, 132025- 132032. doi: 10.1109/ACCESS.2021.3113329
35	NGUYEN T D, NGUYEN D Q, PHUNG D. A novel embedding model for knowledge base completion based on convolutional neural network[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, USA: Association for Computational Linguistics, 2018: 327-333.
36	BALAZEVIC I, ALLEN C, HOSPEDALES T. TuckER: tensor factorization for knowledge graph completion[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Washington D. C., USA: IEEE Press, 2019: 5185-5194.
37	LI L Y, ZHANG X, MA Y B, et al. A knowledge graph completion model based on contrastive learning and relation enhancement method. Knowledge-Based Systems, 2022, 256, 109889. doi: 10.1016/j.knosys.2022.109889
38	ADNAN Z, SUMMAYA S, JUNDE C, et al. Learning knowledge graph embeddings by deep relational roto-reflection. Knowledge-Based Systems, 2022, 252, 109451. doi: 10.1016/j.knosys.2022.109451
39	SOCHER R, CHEN D, MANNING C D, et al. Reasoning with neural tensor networks for knowledge base completion[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. Washington D. C., USA: IEEE Press, 2013: 926-934.
40	MILLER G A. WordNet: a lexical database for English. Communications of the ACM, 1995, 38(11): 39- 41. doi: 10.1145/219717.219748

[1]	李泽霖, 吕兆峰, 陈富强, 李克. 基于多跳信息融合的实体对齐模型[J]. 计算机工程, 2024, 50(9): 142-152.
[2]	汤志康, 武毓琦, 李春英, 汤庸. 基于知识图谱卷积网络的学习资源推荐[J]. 计算机工程, 2024, 50(9): 153-160.
[3]	李华昱, 张智康, 闫阳, 岳阳. 基于知识图谱增强的领域多模态实体识别[J]. 计算机工程, 2024, 50(8): 31-39.
[4]	高爽, 史轶伦, 徐巧枝, 于磊. 基于对比学习的非对称编解码结构的心脏MRI分割研究[J]. 计算机工程, 2024, 50(8): 290-300.
[5]	陈宇航, 杨勇, 先木斯亚·买买提明, 帕力旦·吐尔逊, 樊小超, 任鸽, 刁宇峰. 基于主题感知和语义增强的作文自动评分方法[J]. 计算机工程, 2024, 50(8): 363-371.
[6]	李惠乾, 钟柏昌. 教育知识图谱: 研究进展与未来发展——基于2013—2023年中文核心期刊载文的分析[J]. 计算机工程, 2024, 50(7): 1-12.
[7]	王炼红, 林飞鹏, 李潇瑶, 谌桂枝, 周莉. 融入课程知识图谱的KMAKT预测[J]. 计算机工程, 2024, 50(7): 23-31.
[8]	钱来, 赵卫伟. 基于对比学习和注意力机制的文本分类方法[J]. 计算机工程, 2024, 50(7): 104-111.
[9]	陈佳玉, 王元龙, 张虎. 基于文本知识增强的问题生成模型[J]. 计算机工程, 2024, 50(6): 86-93.
[10]	武星, 殷浩宇, 姚骏峰, 李卫民, 钱权. 面向视频数据的多模态情感分析[J]. 计算机工程, 2024, 50(6): 218-227.
[11]	江惠珍, 孙艳春, 黄罡. 基于知识图谱的GitHub层次化学习和检索服务[J]. 计算机工程, 2024, 50(5): 16-25.
[12]	张宝鑫, 杨丹, 聂铁铮, 寇月. 基于自监督的多视角图协同过滤推荐方法[J]. 计算机工程, 2024, 50(5): 100-110.
[13]	李田芳, 普园媛, 赵征鹏, 徐丹, 钱文华. 基于CLIP和双空间自适应归一化的图像翻译[J]. 计算机工程, 2024, 50(5): 229-240.
[14]	隗昊, 刁宏悦, 孔亮宸, 邓耀臣. 东北亚舆情文本细粒度命名实体识别方法研究[J]. 计算机工程, 2024, 50(5): 354-362.
[15]	隗昊, 刁宏悦, 孔亮宸, 邓耀臣. 东北亚舆情文本细粒度命名实体识别方法研究[J]. 计算机工程, 2024, 50(5): 354-362.

选择文件类型/文献管理软件名称

选择包含的内容