作者投稿和查稿 主编审稿 专家审稿 编委审稿 远程编辑

计算机工程 ›› 2023, Vol. 49 ›› Issue (3): 73-79. doi: 10.19678/j.issn.1000-3428.0062149

• 人工智能与模式识别 • 上一篇    下一篇

利用知识强化语言模型的口语理解方法

刘高军1,2,3, 王岳1,2,3, 段建勇1,2,3, 何丽1,2,3, 王昊1,2,3   

  1. 1. 北方工业大学 信息学院, 北京 100144;
    2. CNONIX国家标准应用与推广实验室, 北京 100144;
    3. 富媒体数字出版内容组织与知识服务重点实验室, 北京 100144
  • 收稿日期:2021-07-21 修回日期:2021-12-23 发布日期:2023-03-09
  • 作者简介:刘高军(1962—),男,教授,主研方向为软件工程与服务;王岳,硕士研究生;段建勇,教授、博士;何丽,副教授、硕士;王昊,副教授、博士。
  • 基金资助:
    国家自然科学基金面上项目“面向新闻事件的查询时效性计算模型研究”(61972003);富媒体数字出版内容组织与知识服务重点实验室项目(ZD2021-11/05)。

Methods of Spoken Language Understanding Using Knowledge Reinforcement Language Model

LIU Gaojun1,2,3, WANG Yue1,2,3, DUAN Jianyong1,2,3, HE Li1,2,3, WANG Hao1,2,3   

  1. 1. School of Information Science and Technology, North China University of Technology, Beijing 100144, China;
    2. CNONIX National Standard Application and Promotion Laboratory, Beijing 100144, China;
    3. Rich Media Digital Publishing Content Organization and Knowledge Service Key Laboratory, Beijing 100144, China
  • Received:2021-07-21 Revised:2021-12-23 Published:2023-03-09

摘要: 基于预训练的语言模型在口语理解(SLU)任务中具有优异的性能表现。然而,与人类理解语言的方式相比,单纯的语言模型只能建立文本层级的上下文关联,缺少丰富的外部知识来支持其完成更为复杂的推理。提出一种针对SLU任务的基于Transformer的双向编码器表示(BERT)的联合模型。引入单词级别的意图特征并使用注意力机制为BERT融合外部知识。此外,由于SLU包含意图检测和槽填充2个相互关联的子任务,模型通过联合训练捕捉2个子任务间的关联性,充分运用这种关联性增强外部知识对于SLU任务的性能提升效果,并将外部知识转化为可用于特定子任务的特征信息。在ATIS和Snips 2个公开数据集上的实验结果表明,该模型句子级别的语义准确率分别为89.1%和93.3%,与BERT模型相比,分别提升了0.9和0.4个百分点,能够有效利用外部知识提升自身性能,在SLU任务中拥有比BERT更为优秀的性能表现。

关键词: 口语理解, 外部知识, 语言模型, 意图检测, 槽填充, 联合训练

Abstract: Pretrained language representations have shown excellent performance in Spoken Language Understanding(SLU).However, compared with the way humans understand language, language representations can only establish the contextual relationship of an input sequence.Additionally, they lack the external knowledge required to complete more complex reasoning.This paper proposes a joint model based on the Bidirectional Encoder Representations from Transformer(BERT) for SLU.The model uses the attention mechanism to fuse external knowledge.In addition, SLUs contain two interrelated subtasks, namely intention detection and slot filling.Therefore, the model captures the correlation between the two subtasks through joint training.The model makes full use of this correlation to further enhance the performance improvement effect of the external knowledge on SLU tasks.Additionally, the external knowledge is converted into characteristic information that can be used for specific subtasks.The experimental results on the ATIS and Snips datasets show that the semantic accuracy of the sentence level of this model is increased by 89.1% and 93.3%, respectively.This is 0.9 and 0.4 percentage points higher than that of the BERT model.Additionally, the model can effectively use external knowledge to improve its own performance.Therefore, the model exhibits better performance in SLU missions than BERT.

Key words: Spoken Language Understanding(SLU), external knowledge, language model, intention detection, slot filling, joint training

中图分类号: