利用知识强化语言模型的口语理解方法

doi:10.19678/j.issn.1000-3428.0062149

计算机工程 ›› 2023, Vol. 49 ›› Issue (3): 73-79. doi: 10.19678/j.issn.1000-3428.0062149

利用知识强化语言模型的口语理解方法

刘高军^1,2,3, 王岳^1,2,3, 段建勇^1,2,3, 何丽^1,2,3, 王昊^1,2,3

1. 北方工业大学信息学院, 北京 100144;
2. CNONIX国家标准应用与推广实验室, 北京 100144;
3. 富媒体数字出版内容组织与知识服务重点实验室, 北京 100144

收稿日期:2021-07-21 修回日期:2021-12-23 发布日期:2023-03-09
作者简介:刘高军(1962—),男,教授,主研方向为软件工程与服务;王岳,硕士研究生;段建勇,教授、博士;何丽,副教授、硕士;王昊,副教授、博士。
基金资助:
国家自然科学基金面上项目“面向新闻事件的查询时效性计算模型研究”（61972003）；富媒体数字出版内容组织与知识服务重点实验室项目（ZD2021-11/05）。

Methods of Spoken Language Understanding Using Knowledge Reinforcement Language Model

LIU Gaojun^1,2,3, WANG Yue^1,2,3, DUAN Jianyong^1,2,3, HE Li^1,2,3, WANG Hao^1,2,3

1. School of Information Science and Technology, North China University of Technology, Beijing 100144, China;
2. CNONIX National Standard Application and Promotion Laboratory, Beijing 100144, China;
3. Rich Media Digital Publishing Content Organization and Knowledge Service Key Laboratory, Beijing 100144, China

Received:2021-07-21 Revised:2021-12-23 Published:2023-03-09

摘要/Abstract

摘要： 基于预训练的语言模型在口语理解（SLU）任务中具有优异的性能表现。然而，与人类理解语言的方式相比，单纯的语言模型只能建立文本层级的上下文关联，缺少丰富的外部知识来支持其完成更为复杂的推理。提出一种针对SLU任务的基于Transformer的双向编码器表示（BERT）的联合模型。引入单词级别的意图特征并使用注意力机制为BERT融合外部知识。此外，由于SLU包含意图检测和槽填充2个相互关联的子任务，模型通过联合训练捕捉2个子任务间的关联性，充分运用这种关联性增强外部知识对于SLU任务的性能提升效果，并将外部知识转化为可用于特定子任务的特征信息。在ATIS和Snips 2个公开数据集上的实验结果表明，该模型句子级别的语义准确率分别为89.1%和93.3%，与BERT模型相比，分别提升了0.9和0.4个百分点，能够有效利用外部知识提升自身性能，在SLU任务中拥有比BERT更为优秀的性能表现。

关键词: 口语理解, 外部知识, 语言模型, 意图检测, 槽填充, 联合训练

Abstract: Pretrained language representations have shown excellent performance in Spoken Language Understanding(SLU).However, compared with the way humans understand language, language representations can only establish the contextual relationship of an input sequence.Additionally, they lack the external knowledge required to complete more complex reasoning.This paper proposes a joint model based on the Bidirectional Encoder Representations from Transformer(BERT) for SLU.The model uses the attention mechanism to fuse external knowledge.In addition, SLUs contain two interrelated subtasks, namely intention detection and slot filling.Therefore, the model captures the correlation between the two subtasks through joint training.The model makes full use of this correlation to further enhance the performance improvement effect of the external knowledge on SLU tasks.Additionally, the external knowledge is converted into characteristic information that can be used for specific subtasks.The experimental results on the ATIS and Snips datasets show that the semantic accuracy of the sentence level of this model is increased by 89.1% and 93.3%, respectively.This is 0.9 and 0.4 percentage points higher than that of the BERT model.Additionally, the model can effectively use external knowledge to improve its own performance.Therefore, the model exhibits better performance in SLU missions than BERT.

Key words: Spoken Language Understanding(SLU), external knowledge, language model, intention detection, slot filling, joint training

中图分类号:

TP18

刘高军, 王岳, 段建勇, 何丽, 王昊. 利用知识强化语言模型的口语理解方法[J]. 计算机工程, 2023, 49(3): 73-79.

LIU Gaojun, WANG Yue, DUAN Jianyong, HE Li, WANG Hao. Methods of Spoken Language Understanding Using Knowledge Reinforcement Language Model[J]. Computer Engineering, 2023, 49(3): 73-79.

https://www.ecice06.com/CN/Y2023/V49/I3/73

图/表 10

20230314184711

20230314184714

20230314184717

20230314184720

20230314184723

20230314184726

20230314184729

20230314184732

20230314184735

20230314184739

参考文献

[1] DEVLIN J, CHANG M W, LEE K, et al.BERT:pre-training of deep bidirectional transformers for language understanding[EB/OL].[2021-06-20].https://arxiv.org/abs/1810.04805.
[2] CHEN Q, ZHUO Z, WANG W.BERT for joint intent classification and slot filling[EB/OL].[2021-06-20].https://arxiv.org/abs/1902.10909.
[3] YANG A, WANG Q, LIU J, et al.Enhancing pre-trained language representations with rich knowledge for machine reading comprehension[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Stroudsburg, USA:Association for Computational Linguistics, 2019:218-229.
[4] MILLER G A.WordNet[J].Communications of the ACM, 1995, 38(11):39-41.
[5] CARLSON A, BETTERIDGE J, KISIEL B.Toward an architecture for never-ending language learning[C]//Proceedings of the 24th AAAI Conference on Artificial Intelligence.[S.1.]:AAAI Press, 2010:365-379.
[6] YANG B S, MITCHELL T.Leveraging knowledge bases in LSTMs for improving machine reading[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.Washington D.C., USA:IEEE Press, 2017:332-346.
[7] GUO D, TUR G, YIH W T, et al.Joint semantic utterance classification and slot filling with recursive neural networks[C]//Proceedings of IEEE Spoken Language Technology Workshop.Washington D.C., USA:IEEE Press, 2015:554-559.
[8] XU P Y, SARIKAYA R.Convolutional neural network based triangular CRF for joint intent detection and slot filling[C]//Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding.Washington D.C., USA:IEEE Press, 2014:78-83.
[9] LIU B, LANE I.Attention-based recurrent neural network models for joint intent detection and slot filling[EB/OL].[2021-06-20].https://arxiv.org/abs/1609.01454.
[10] ZHANG X D, WANG H F.A joint model of intent determination and slot filling for spoken language understanding[C]//Proceedings of the 25th International Joint Conference on Artificial Intelligence.New York, USA:ACM Press, 2016:2993-2999.
[11] GOO C W, GAO G, HSU Y K, et al.Slot-gated modeling for joint slot filling and intent prediction[C]//Proceedings of 2018 Conference of the North American Chapter of the Association for Computational Linguistic.Washington D.C., USA:IEEE Press, 2018:1257-1266.
[12] HAIHONG E, NIU P Q, CHEN Z F, et al.A novel bi-directional interrelated model for joint intent detection and slot filling[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Washington D.C., USA:IEEE Press, 2019:457-471.
[13] QIN L B, CHE W X, LI Y M, et al.A stack-propagation framework with token-level intent detection for spoken language understanding[C]//Processing of the 9th International Joint Conference on Natural Language Processing.Washington D.C., USA:IEEE Press, 2019:753-766.
[14] ZHANG Y, WEISS D.Stack-propagation:improved representation learning for syntax[EB/OL].[2021-06-20].https://arxiv.org/abs/1603.06598.
[15] HEMPHILL C T, GODFREY J J, DODDINGTON G R.The atis spoken language systems pilot corpus[EB/OL].[2021-06-20].https://aclanthology.org/H90-1021/.
[16] COUCKE A, SAADE A, BALL A, et al.Snips voice platform:an embedded spoken language understanding system for private-by-design voice interfaces[EB/OL].[2021-06-20].https://arxiv.org/abs/1805.10190.
[17] HOCHREITER S, SCHMIDHUBER J.Long short-term memory[J].Neural Computation, 1997, 9(8):1735-1780.
[18] ZHONG V, XIONG C, SOCHER R.Global-locally self-attentive dialogue state tracker[EB/OL].[2021-06-20].https://arxiv.org/abs/1805.09655.
[19] VASWANI A, SHAZEER N, PARMAR N, et al.Attention is all you need[C]//Proceedings of the 31th Annual Conference on Neural Information Processing Systems.Cambridge, USA:MIT Press, 2017:367-378.
[20] BA J L, KIROS J R, HINTON G E.Layer normalization[EB/OL].[2021-06-20].https://arxiv.org/abs/1607.06450.
[21] KINGMA D P, BA J.Adam:a method for stochastic optimization[EB/OL].[2021-06-20].https://arxiv.org/abs/1412.6980.
[22] ZAREMBA W, SUTSKEVER I, VINYALS O.Recurrent neural network regularization[EB/OL].[2021-06-20].https://arxiv.org/abs/1409.2329.
[23] KIM Y.Convolutional neural networks for sentence classification[EB/OL].[2021-06-20].https://arxiv.org/abs/1408.5882.
[24] LAFFERTY J, MCCALLUM A, PEREIRA F C N.Conditional random fields:probabilistic models for segmenting and labeling sequence data[EB/OL].[2021-06-20].https://repository.upenn.edu/cgi/viewcontent.cgi?article=1162&context=cis_papers.
[25] BURGES C J, BURGES C.A tutorial on support vector machines for pattern recognition[EB/OL].[2021-06-20].https://www.microsoft.com/en-us/research/publication/a-tutorial-on-support-vector-machines-for-pattern-recognition/?from=http%3A%2F%2Fresearch.microsoft.com%2Fpubs%2F67119%2Fsvmtutorial.pdf.

选择文件类型/文献管理软件名称

选择包含的内容

利用知识强化语言模型的口语理解方法

Methods of Spoken Language Understanding Using Knowledge Reinforcement Language Model

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	罗焕坤, 葛一烽, 刘帅. 大语言模型在数学推理中的研究进展[J]. 计算机工程, 2024, 50(9): 1-17.
[2]	杨冬菊, 黄俊涛. 基于大语言模型的中文科技文献标注方法[J]. 计算机工程, 2024, 50(9): 113-120.
[3]	陈宇航, 杨勇, 先木斯亚·买买提明, 帕力旦·吐尔逊, 樊小超, 任鸽, 刁宇峰. 基于主题感知和语义增强的作文自动评分方法[J]. 计算机工程, 2024, 50(8): 363-371.
[4]	刘娟, 段友祥, 陆誉翕, 张鲁. 引入知识增强和对比学习的知识图谱补全[J]. 计算机工程, 2024, 50(7): 112-122.
[5]	曾碧卿, 陈鹏飞, 姚勇涛. 融合思维链和低秩自适应微调的方面情感三元组抽取[J]. 计算机工程, 2024, 50(7): 53-62.
[6]	翟洁, 李艳豪, 李彬彬, 郭卫斌. 基于大语言模型的个性化实验报告评语自动生成与应用[J]. 计算机工程, 2024, 50(7): 42-52.
[7]	翟洁, 李艳豪, 李彬彬, 郭卫斌. 基于大语言模型的个性化实验报告评语自动生成与应用[J]. 计算机工程, 2024, 50(7): 42-52.
[8]	杨兴睿, 马斌, 李森垚, 钟忺. 基于大语言模型的教育文本幂等摘要方法[J]. 计算机工程, 2024, 50(7): 32-41.
[9]	陈佳玉, 王元龙, 张虎. 基于文本知识增强的问题生成模型[J]. 计算机工程, 2024, 50(6): 86-93.
[10]	隗昊, 刁宏悦, 孔亮宸, 邓耀臣. 东北亚舆情文本细粒度命名实体识别方法研究[J]. 计算机工程, 2024, 50(5): 354-362.
[11]	隗昊, 刁宏悦, 孔亮宸, 邓耀臣. 东北亚舆情文本细粒度命名实体识别方法研究[J]. 计算机工程, 2024, 50(5): 354-362.
[12]	张洪程, 李林育, 杨莉, 伞晨峻, 尹春林, 颜冰, 于虹, 张璇. 基于对比学习与语言模型增强嵌入的知识图谱补全[J]. 计算机工程, 2024, 50(4): 168-176.
[13]	李敬灿, 肖萃林, 覃晓婷, 谢夏. 基于大语言模型与语义增强的文本关系抽取算法[J]. 计算机工程, 2024, 50(4): 87-94.
[14]	侯钰涛, 阿布都克力木·阿布力孜, 史亚庆, 马依拉木·木斯得克, 哈里旦木·阿布都克里木. 面向"一带一路"的低资源语言机器翻译研究[J]. 计算机工程, 2024, 50(4): 332-341.
[15]	施竣潇, 陈艳平, 穆肇南. 融合多尺度跨度特征的谓语中心词识别模型[J]. 计算机工程, 2024, 50(10): 137-144.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

利用知识强化语言模型的口语理解方法

Methods of Spoken Language Understanding Using Knowledge Reinforcement Language Model

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献

相关文章 15

编辑推荐

Metrics

本文评价