计算机工程 ›› 2021, Vol. 47 ›› Issue (9): 18-33.doi: 10.19678/j.issn.1000-3428.0060823

• 热点与综述 • 上一篇    下一篇

知识感知的预训练语言模型综述

李瑜泽1,2,3, 栾馨1,2,3, 柯尊旺2,3,4, 李哲2,3,4, 吾守尔·斯拉木1,2,3   

  1. 1. 新疆大学 信息科学与工程学院, 乌鲁木齐 830046;
    2. 新疆多语种信息技术实验室, 乌鲁木齐 830046;
    3. 新疆多语种信息技术研究中心, 乌鲁木齐 830046;
    4. 新疆大学 软件学院, 乌鲁木齐 830046
  • 收稿日期:2021-02-05 修回日期:2021-03-31 发布日期:2021-04-06
  • 作者简介:李瑜泽(1997-),女,硕士研究生,主研方向为自然语言处理、舆情分析;栾馨,硕士研究生;柯尊旺(通信作者),讲师、博士;李哲,硕士研究生;吾守尔·斯拉木,教授、博士生导师。
  • 基金项目:
    国家重点研发计划(2017YFC0820702-3)。

Survey of Knowledge-Aware Pre-Trained Language Models

LI Yuze1,2,3, LUAN Xin1,2,3, KE Zunwang2,3,4, LI Zhe2,3,4, Wushour Silamu1,2,3   

  1. 1. College of Information Science and Engineering, Xinjiang University, Urumqi 830046, China;
    2. Xinjiang Laboratory of Multi-Language Information Technology, Urumqi 830046, China;
    3. Xinjiang Multi-Language Information Technology Research Center, Urumqi 830046, China;
    4. School of Software, Xinjiang University, Urumqi 830046, China
  • Received:2021-02-05 Revised:2021-03-31 Published:2021-04-06

摘要: 随着自然语言处理(NLP)领域中预训练技术的快速发展,将外部知识引入到预训练语言模型的知识驱动方法在NLP任务中表现优异,知识表示学习和预训练技术为知识融合的预训练方法提供了理论依据。概述目前经典预训练方法的相关研究成果,分析在新兴预训练技术支持下具有代表性的知识感知的预训练语言模型,分别介绍引入不同外部知识的预训练语言模型,并结合相关实验数据评估知识感知的预训练语言模型在NLP各个下游任务中的性能表现。在此基础上,分析当前预训练语言模型发展过程中所面临的问题和挑战,并对领域发展前景进行展望。

关键词: 自然语言处理, 知识表征, 语义知识, 预训练, 语言模型

Abstract: In the field of Natural Language Processing(NLP), the recent years has witnessed a rapid development in pre-training technology, and the knowledge-driven method that injects external knowledge into a pre-trained language model performs excellently in NLP tasks.The techniques of knowledge representation learning and pre-training provide theoretical foundation for the knowledge-based pre-training method.This paper briefly introduces the development of the classical pre-trained methods.Then it analyzes the representative knowledge-aware pre-trained language models supported by new pre-training technology.According to the types of external knowledge, this paper introduces the pre-trained language models injected with different external knowledge.Based on relevant experimental data, it subsequently evaluates the performance of the knowledge-aware pre-trained language models in various downstream tasks of NLP.On this basis, the paper analyzes the problems and challenges faced by the developing pre-trained language models, and discusses the development trends of this field.

Key words: Natural Language Processing(NLP), knowledge representation, semantic knowledge, pre-training, language model

中图分类号: