Methods of Spoken Language Understanding Using Knowledge Reinforcement Language Model

doi:10.19678/j.issn.1000-3428.0062149

Abstract

Abstract: Pretrained language representations have shown excellent performance in Spoken Language Understanding(SLU).However, compared with the way humans understand language, language representations can only establish the contextual relationship of an input sequence.Additionally, they lack the external knowledge required to complete more complex reasoning.This paper proposes a joint model based on the Bidirectional Encoder Representations from Transformer(BERT) for SLU.The model uses the attention mechanism to fuse external knowledge.In addition, SLUs contain two interrelated subtasks, namely intention detection and slot filling.Therefore, the model captures the correlation between the two subtasks through joint training.The model makes full use of this correlation to further enhance the performance improvement effect of the external knowledge on SLU tasks.Additionally, the external knowledge is converted into characteristic information that can be used for specific subtasks.The experimental results on the ATIS and Snips datasets show that the semantic accuracy of the sentence level of this model is increased by 89.1% and 93.3%, respectively.This is 0.9 and 0.4 percentage points higher than that of the BERT model.Additionally, the model can effectively use external knowledge to improve its own performance.Therefore, the model exhibits better performance in SLU missions than BERT.

Key words: Spoken Language Understanding(SLU), external knowledge, language model, intention detection, slot filling, joint training

摘要： 基于预训练的语言模型在口语理解（SLU）任务中具有优异的性能表现。然而，与人类理解语言的方式相比，单纯的语言模型只能建立文本层级的上下文关联，缺少丰富的外部知识来支持其完成更为复杂的推理。提出一种针对SLU任务的基于Transformer的双向编码器表示（BERT）的联合模型。引入单词级别的意图特征并使用注意力机制为BERT融合外部知识。此外，由于SLU包含意图检测和槽填充2个相互关联的子任务，模型通过联合训练捕捉2个子任务间的关联性，充分运用这种关联性增强外部知识对于SLU任务的性能提升效果，并将外部知识转化为可用于特定子任务的特征信息。在ATIS和Snips 2个公开数据集上的实验结果表明，该模型句子级别的语义准确率分别为89.1%和93.3%，与BERT模型相比，分别提升了0.9和0.4个百分点，能够有效利用外部知识提升自身性能，在SLU任务中拥有比BERT更为优秀的性能表现。

关键词: 口语理解, 外部知识, 语言模型, 意图检测, 槽填充, 联合训练

CLC Number:

TP18

LIU Gaojun, WANG Yue, DUAN Jianyong, HE Li, WANG Hao. Methods of Spoken Language Understanding Using Knowledge Reinforcement Language Model[J]. Computer Engineering, 2023, 49(3): 73-79.

刘高军, 王岳, 段建勇, 何丽, 王昊. 利用知识强化语言模型的口语理解方法[J]. 计算机工程, 2023, 49(3): 73-79.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0062149

http://www.ecice06.com/EN/Y2023/V49/I3/73

Figures/Tables 10

References

[1] DEVLIN J, CHANG M W, LEE K, et al.BERT:pre-training of deep bidirectional transformers for language understanding[EB/OL].[2021-06-20].https://arxiv.org/abs/1810.04805.
[2] CHEN Q, ZHUO Z, WANG W.BERT for joint intent classification and slot filling[EB/OL].[2021-06-20].https://arxiv.org/abs/1902.10909.
[3] YANG A, WANG Q, LIU J, et al.Enhancing pre-trained language representations with rich knowledge for machine reading comprehension[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Stroudsburg, USA:Association for Computational Linguistics, 2019:218-229.
[4] MILLER G A.WordNet[J].Communications of the ACM, 1995, 38(11):39-41.
[5] CARLSON A, BETTERIDGE J, KISIEL B.Toward an architecture for never-ending language learning[C]//Proceedings of the 24th AAAI Conference on Artificial Intelligence.[S.1.]:AAAI Press, 2010:365-379.
[6] YANG B S, MITCHELL T.Leveraging knowledge bases in LSTMs for improving machine reading[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.Washington D.C., USA:IEEE Press, 2017:332-346.
[7] GUO D, TUR G, YIH W T, et al.Joint semantic utterance classification and slot filling with recursive neural networks[C]//Proceedings of IEEE Spoken Language Technology Workshop.Washington D.C., USA:IEEE Press, 2015:554-559.
[8] XU P Y, SARIKAYA R.Convolutional neural network based triangular CRF for joint intent detection and slot filling[C]//Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding.Washington D.C., USA:IEEE Press, 2014:78-83.
[9] LIU B, LANE I.Attention-based recurrent neural network models for joint intent detection and slot filling[EB/OL].[2021-06-20].https://arxiv.org/abs/1609.01454.
[10] ZHANG X D, WANG H F.A joint model of intent determination and slot filling for spoken language understanding[C]//Proceedings of the 25th International Joint Conference on Artificial Intelligence.New York, USA:ACM Press, 2016:2993-2999.
[11] GOO C W, GAO G, HSU Y K, et al.Slot-gated modeling for joint slot filling and intent prediction[C]//Proceedings of 2018 Conference of the North American Chapter of the Association for Computational Linguistic.Washington D.C., USA:IEEE Press, 2018:1257-1266.
[12] HAIHONG E, NIU P Q, CHEN Z F, et al.A novel bi-directional interrelated model for joint intent detection and slot filling[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Washington D.C., USA:IEEE Press, 2019:457-471.
[13] QIN L B, CHE W X, LI Y M, et al.A stack-propagation framework with token-level intent detection for spoken language understanding[C]//Processing of the 9th International Joint Conference on Natural Language Processing.Washington D.C., USA:IEEE Press, 2019:753-766.
[14] ZHANG Y, WEISS D.Stack-propagation:improved representation learning for syntax[EB/OL].[2021-06-20].https://arxiv.org/abs/1603.06598.
[15] HEMPHILL C T, GODFREY J J, DODDINGTON G R.The atis spoken language systems pilot corpus[EB/OL].[2021-06-20].https://aclanthology.org/H90-1021/.
[16] COUCKE A, SAADE A, BALL A, et al.Snips voice platform:an embedded spoken language understanding system for private-by-design voice interfaces[EB/OL].[2021-06-20].https://arxiv.org/abs/1805.10190.
[17] HOCHREITER S, SCHMIDHUBER J.Long short-term memory[J].Neural Computation, 1997, 9(8):1735-1780.
[18] ZHONG V, XIONG C, SOCHER R.Global-locally self-attentive dialogue state tracker[EB/OL].[2021-06-20].https://arxiv.org/abs/1805.09655.
[19] VASWANI A, SHAZEER N, PARMAR N, et al.Attention is all you need[C]//Proceedings of the 31th Annual Conference on Neural Information Processing Systems.Cambridge, USA:MIT Press, 2017:367-378.
[20] BA J L, KIROS J R, HINTON G E.Layer normalization[EB/OL].[2021-06-20].https://arxiv.org/abs/1607.06450.
[21] KINGMA D P, BA J.Adam:a method for stochastic optimization[EB/OL].[2021-06-20].https://arxiv.org/abs/1412.6980.
[22] ZAREMBA W, SUTSKEVER I, VINYALS O.Recurrent neural network regularization[EB/OL].[2021-06-20].https://arxiv.org/abs/1409.2329.
[23] KIM Y.Convolutional neural networks for sentence classification[EB/OL].[2021-06-20].https://arxiv.org/abs/1408.5882.
[24] LAFFERTY J, MCCALLUM A, PEREIRA F C N.Conditional random fields:probabilistic models for segmenting and labeling sequence data[EB/OL].[2021-06-20].https://repository.upenn.edu/cgi/viewcontent.cgi?article=1162&context=cis_papers.
[25] BURGES C J, BURGES C.A tutorial on support vector machines for pattern recognition[EB/OL].[2021-06-20].https://www.microsoft.com/en-us/research/publication/a-tutorial-on-support-vector-machines-for-pattern-recognition/?from=http%3A%2F%2Fresearch.microsoft.com%2Fpubs%2F67119%2Fsvmtutorial.pdf.

Please choose a citation manager

Content to export