基于单向Transformer和孪生网络的多轮任务型对话技术

doi:10.19678/j.issn.1000-3428.0058557

摘要/Abstract

摘要： 循环神经网络和Transformer在多轮对话系统的建模上依赖大量的样本数据且回复准确率过低。为此，提出一种针对任务型对话系统的建模方法。引入预训练模型对句子语意和对话过程进行深度编码，对Transformer模型进行精简，仅保留编码器部分的单向Transformer，将应答部分抽象成不同的指令，采用孪生网络对指令进行相似度排序，选择相似度最高的指令生成应答。在MultiWOZ数据集上的实验结果表明，与LSTM和基于Transformer模型相比，该方法预测速度更快，在小数据集上具有更好的性能，在大数据集上也能取得与当前先进模型相当的效果。

关键词: 循环神经网络, 多轮对话系统, 预训练模型, Transformer模型, 孪生网络

Abstract: The existing Recurrent Neural Network(RNN) and Transformer models rely on a large amount of sample data for the modeling of the multi-turn dialogue system,and the accuracy of answering is low.To address the problem,a new modeling method for the task-oriented dialogue system is proposed.Some pre-trained models are introduced for deep encoding of the sentence semantics and the dialog contents.At the same time,the Transformer model is simplified to a unidirectional transformer with only the encoder retained.On this basis,the answering part is abstracted to different commands,which are sorted based on similarity by using the siamese network.The command with the highest similarity is chosen to generate the answer.The experimental results on the MultiWOZ dataset show that compared to LSTM and Transformer-based models,the proposed method has a faster prediction speed,providing better performance on small datasets and equal performance on large datasets.

Key words: Recurrent Neural Network(RNN), multi-turn dialogue system, pre-training model, Transformer model, siamese network

中图分类号:

TP18

王涛, 刘超辉, 郑青青, 黄嘉曦. 基于单向Transformer和孪生网络的多轮任务型对话技术[J]. 计算机工程, 2021, 47(7): 55-58,66.

WANG Tao, LIU Chaohui, ZHENG Qingqing, HUANG Jiaxi. Multi-turn Task-oriented Dialogue Technology Based on Unidirectional Transformer and Siamese Network[J]. Computer Engineering, 2021, 47(7): 55-58,66.

https://www.ecice06.com/CN/Y2021/V47/I7/55

图/表 18

20210720203127

20210720203131

20210720203136

20210720203140

20210720203145

20210720203149

20210720203153

20210720203157

20210720203202

20210720203206

20210720203211

20210720203216

20210720203220

20210720203225

20210720203230

20210720203233

20210720203238

20210720203242

参考文献

[1] 张伟男,刘挺.聊天机器人技术研究进展[J].中国人工智能学会通讯,2016,6(1):17-21. ZHANG W N,LIU T.The research progress of chatbot technology[J].Chinese Artificial Intelligence Society Newsletter,2016,6(1):17-21.(in Chinese)
[2] CHEN H S,LIU X R,YIN D W,et al.A survey on dialogue systems:recent advances and new frontiers[J].ACM SIGKDD Explorations Newsletter,2017,19(2):25-35.
[3] HUANG Y F,LI Z C,ZHANG Z S,et al.Moon IME:neural-based Chinese pinyin aided input method with customizable association[C]//Proceedings of the 56th IEEE Annual Meeting of the Association for Computational Linguistics-System Demonstrations.Washington D.C.,USA:IEEE Press,2018:140-145.
[4] TURING A M.Computing machinery and intelligence[J].Mind,1950,59:433-460.
[5] BENGIO Y,DUCCHARME R,VINCENT P,et al.A neural probabilistic language model[J].Journal of Machine Learning Research,2003,3(2):1137-1155.
[6] MIKOLOV T,KARAFIAT M,BURGET L,et al.Recurrent neural network based language model[C]//Proceedings of IEEE INTERSPEECH'10.Washington D.C.,USA:IEEE Press,2010:1045-1048.
[7] HOCHREITER S,SCHMIDHUBER J.Long short-term memory[J].Neural Computation,1997,9(8):1735-1780.
[8] HOSSEINI S A,BAZRAFKAN S,VATANDOOST H,et al.The insecticidal effect of diatomaceous earth against adults and nymphs of Blattella Germanica[J].Asian Pacific Journal of Tropical Biomedicine,2014,4(1):228-232.
[9] VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[EB/OL].[2020-05-05].https://arxiv.org/abs/1706.03762.
[10] DEVLIN J,CHANG M W,LEE K,et al.BERT:pre-training of deep bidirectional transformers for language understanding[EB/OL].[2020-05-05].https://arxiv.org/abs/1810.04805.
[11] YAN R,ZHAO D Y.Coupled context modeling for deep chit-chat:towards conversations between human and computer[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.New York,USA:ACM Press,2018:2574-2583.
[12] CHEN H,LIU X R,YIN D W,et al.A survey on dialogue systems:recent advances and new frontiers[EB/OL].[2020-05-05].https://arxiv.org/abs/1711.01731.
[13] QIU X P,SUN T X,XU Y G,et al.Pre-trained models for natural language processing:a survey[J].Science China Technological Sciences,2020,63(10):1872-1897.
[14] CHO K,VAN MERRIENBOER B,GULCEHRE C,et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[EB/OL].[2020-05-05].https://arxiv.org/abs/1406.1078.
[15] CHOPRA S,HADSELL R,LECUN Y.Learning a similarity metric discriminatively,with application to face verification[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2005:1063-6919.
[16] ZHOU X Y,DONG D X,WU H,et al.Multi-view response selection for human-computer conversation[C]//Proceedings of IEEE Conference on Empirical Methods in Natural Language Processing.Washington D.C.,USA:IEEE Press,2016:372-381.
[17] LIPTON Z C.A critical review of recurrent neural networks for sequence learning[EB/OL].[2020-05-05].https://arxiv.org/abs/1506.00019.
[18] HENDERSON M,VULIĆ I,GERZ D,et al.Training neural response selection for task-oriented dialogue systems[EB/OL].[2020-05-05].https://arxiv.org/abs/1906.01543.
[19] DINAN E,ROLLER S,SHUSTER K,et al.Wizard of wikipedia:knowledge-powered conversational agents[EB/OL].[2020-05-05].https://arxiv.org/abs/1811.01241.
[20] CHRISTOPHER D M,SURDEANU M,BAUER J,et al.The stanford CoreNLP natural language processing toolkit[C]//Proceedings of the 52nd IEEE Annual Meeting of the Association for Computational Linguistics:System Demonstrations.Washington D.C.,USA:IEEE Press,2014:55-60.
[21] PARK S H,KIM B,KANG C M,et al.Sequence-to-sequence prediction of vehicle trajectory via LSTM encoder-decoder architecture[EB/OL].[2020-05-05].https://arxiv.org/abs/1802.06338.
[22] ERIC M,GOEL R,PAUL S,et al.MultiWOZ 2.1:a consolidated multi-domain dialogue dataset with state corrections and state tracking baselines[EB/OL].[2020-05-05].https://arxiv.org/abs/1907.01669.
[23] YANG Y,YIH W T,MEEK C.WIKIQA:a challenge dataset for open-domain question answering[C]//Proceedings of IEEE EMNLP'15.Washington D.C.,USA:IEEE Press,2015:2013-2018.
[24] LI Y R,SU H,SHEN X Y,et al.DailyDialog:a manually labelled multi-turn dialogue dataset[EB/OL].https://arxiv.org/abs/1710.03957.
[25] MIKOLOV T,SUTSKEVER I,SUTSKEVER I,et al.Distributed representations of words and phrases and their compositionality[EB/OL].[2020-05-05].https://arxiv.org/abs/1310.4546.

选择文件类型/文献管理软件名称

选择包含的内容