一种注意力增强的自然语言推理模型

doi:10.19678/j.issn.1000-3428.0054953

计算机工程 ›› 2020, Vol. 46 ›› Issue (7): 91-97. doi: 10.19678/j.issn.1000-3428.0054953

一种注意力增强的自然语言推理模型

李冠宇^a,b, 张鹏飞^a,b, 贾彩燕^a,b

北京交通大学 a. 计算机与信息技术学院;b. 交通数据分析与挖掘北京市重点实验室, 北京 100044

收稿日期:2019-05-20 修回日期:2019-08-09 发布日期:2019-08-20
作者简介:李冠宇(1993-),男,硕士研究生,主研方向为自然语言处理、机器学习;张鹏飞,硕士研究生;贾彩燕,教授、博士。
基金资助:
国家自然科学基金（61876016）；中央高校基本科研业务费专项资金（2017JBM023）。

An Attention-enchanced Natural Language Reasoning Model

LI Guanyu^a,b, ZHANG Pengfei^a,b, JIA Caiyan^a,b

a. School of Computer and Information Technology;b. Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University, Beijing 100044, China

Received:2019-05-20 Revised:2019-08-09 Published:2019-08-20

摘要/Abstract

摘要： 在自然语言处理任务中使用注意力机制可准确衡量单词重要度。为此，提出一种注意力增强的自然语言推理模型aESIM。将词注意力层以及自适应方向权重层添加到ESIM模型的双向LSTM网络中，从而更有效地学习单词与句子表示，同时提高前提与假设文本之间局部推理的建模效率。在SNLI、MultiNLI及Quora数据集上的实验结果表明，与ESIM、HBMP、SSE等模型相比，aESIM模型的准确率能够提升0.5%~1%。

关键词: 自然语言处理, 自然语言推理, ESIM模型, 注意力机制, 双向LSTM网络

Abstract: In natural language processing tasks,the attention mechanism can be used to evaluate the importance of a word.On this basis,this paper proposes an attention-enhanced natural language reasoning model,aESIM.The model adds the word attention layer and the adaptive direction weight layer to the bidirectional LSTM network of the ESIM model,so as to learn the representation of words and sentences more effectively,and increase the modelling efficiency of local inference between premises and hypothetical texts.Experimental results on datasets of SNLI,MultiNLI and Quora show that,compared with ESIM,HBMP,SSE and other models,aESIM increases the accuracy rate by 0.5%~1%.

Key words: natural language processing, natural language reasoning, ESIM model, attention mechanism, bidirectional LSTM network

中图分类号:

TP18

李冠宇, 张鹏飞, 贾彩燕. 一种注意力增强的自然语言推理模型[J]. 计算机工程, 2020, 46(7): 91-97.

LI Guanyu, ZHANG Pengfei, JIA Caiyan. An Attention-enchanced Natural Language Reasoning Model[J]. Computer Engineering, 2020, 46(7): 91-97.

https://www.ecice06.com/CN/Y2020/V46/I7/91

参考文献

[1] ZHANG Zhichang,YAO Dongren,LIU Xie,et al.Textual entailment recognition fused with syntactic structure transformation and lexical semantic features[J].Computer Engineering,2015,41(9):199-204.(in Chinese)张志昌,姚东任,刘霞,等.融合句法结构变换与词汇语义特征的文本蕴涵识别[J].计算机工程,2015,41(9):199-204.
[2] LAN Wuwei,XU Wei.Neural network models for paraphrase identification,semantic textual similarity,natural language inference,and question answering[C]//Proceedings of the 27th International Conference on Computational Linguistics.Santa Fe,USA:[s.n.],2018:3890-3902.
[3] GUO Maosheng,ZHANG Yu,LIU Ting.Research advances and prospect of recognizing textual entailment and knowledge acquisition[J].Chinese Journal of Computers,2017,40(4):119-140.(in Chinese)郭茂盛,张宇,刘挺.文本蕴含关系识别与知识获取研究进展及展望[J].计算机学报,2017,40(4):119-140.
[4] BOWMAN S R,ANGELI G,POTTS C,et al.A large annotated corpus for learning natural language inference[J].Empirical Methods in Natural Language Processing,2015,41:632-642.
[5] YANG Z,YANG D,DYER C,et al.Hierarchical attention networks for document classification[EB/OL].[2019-04-10].https://arxiv.org/abs/1707.00896v1.
[6] CHEN Qian,ZHU Xiaodan,LING Zhenhua,et al.Enhanced LSTM for natural language inference[C]//Proceedings of IEEE Meeting on Association for Computational Linguistics.Washington D.C.,USA:IEEE Press,2017:1657-1668.
[7] NIE Y,BANSAL M.Shortcut-stacked sentence encoders for multi-domain inference[C]//Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP.Washington D.C.,USA:IEEE Press,2017:165-178.
[8] CONNEAU A,KIELA D,SCHWENK H,et al.Supervised learning of universal sentence representations from natural language inference data[EB/OL].[2019-04-10].https://arxiv.org/abs/1705.02364v5.
[9] TALMAN A,YLIJYRA A,TIEDEMANN J,et al.Natural language inference with hierarchical BiLSTM max pooling architecture[EB/OL].[2019-04-10].https://arxiv.org/abs/1808.08762v1.
[10] IM J,CHO S.Distance-based self-attention network for natural language inference[EB/OL].[2019-04-10].https://arxiv.org/abs/1712.02047v1.
[11] SHEN Tao,ZHOU Tianyi,LONG Guodong,et al.Reinforced self-attention network:a hybrid of hard and soft attention for sequence modeling[EB/OL].[2019-04-10].https://arxiv.org/abs/1801.10296.
[12] CHENG J,DONG L,LAPATA M,et al.Long short-term memory-networks for machine reading[EB/OL].[2019-04-10].https://arxiv.org/abs/1601.06733.
[13] CHUNG J,GULCEHRE C,CHO K,et al.Empirical evaluation of gated recurrent neural networks on sequence modeling[EB/OL].[2019-04-10].https://arxiv.org/abs/1412.3555.
[14] PARIKH A P,TACKSTROM O,DAS D,et al.A decomposable attention model for natural language inference[EB/OL].[2019-04-10].https://arxiv.org/abs/1606.01933v2.
[15] WANG Z,HAMZA W,FLORIAN R,et al.Bilateral multi-perspective matching for natural language sentences[C]//Proceedings of IEEE International Joint Conference on Artificial Intelligence.Washington D.C.,USA:IEEE Press,2017:4144-4150.
[16] KIM S,KANG I,KWAK N.Semantic sentence matching with densely-connected recurrent and co-attentive information[EB/OL].[2019-04-10].https://arxiv.org/abs/1805.11360.
[17] BAHDANAU D,CHO K,BENGIO Y.Neural machine translation by jointly learning to align and translate[EB/OL].[2019-04-10].https://arxiv.org/abs/1409.0473.
[18] HE H,LIN J J.Pairwise word interaction modeling with deep neural networks for semantic similarity measurement[C]//Proceedings of 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.San Diego,USA:[s.n.],2016:937-948.
[19] WILLIAMS A,NANGIA N,BOEMAN S R,et al.A broad-coverage challenge corpus for sentence understanding through inference[EB/OL].[2019-04-10].https://arxiv.org/abs/1704.05426v2.
[20] PENNINGTON J,SOCHER R,MANNING C D,et al.GloVe:global vectors for word representation[EB/OL].[2019-04-10].https://www.aclweb.org/anthology/D14-1162.

选择文件类型/文献管理软件名称

选择包含的内容

一种注意力增强的自然语言推理模型

An Attention-enchanced Natural Language Reasoning Model

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	王炼红, 林飞鹏, 李潇瑶, 谌桂枝, 周莉. 融入课程知识图谱的KMAKT预测[J]. 计算机工程, 2024, 50(7): 23-31.
[2]	钱来, 赵卫伟. 基于对比学习和注意力机制的文本分类方法[J]. 计算机工程, 2024, 50(7): 104-111.
[3]	刘建敏, 林晖, 汪晓丁. 基于图注意力机制的无地图场景轨迹预测方法[J]. 计算机工程, 2024, 50(7): 144-153.
[4]	屠乃威, 焦猛, 阎馨. 复杂环境下输电线路鸟巢目标图像检测模型[J]. 计算机工程, 2024, 50(7): 216-226.
[5]	肖慈, 徐杨, 张永丹, 冯明文, 黄易仟. 结合注意力和低光增强的夜间语义分割[J]. 计算机工程, 2024, 50(7): 271-281.
[6]	张锡英, 孙守东, 于海浩, 边继龙. 基于空间传播的多视图三维重建[J]. 计算机工程, 2024, 50(7): 293-302.
[7]	贵向泉, 刘世清, 李立, 秦庆松, 李唐艳. 基于改进YOLOv8的景区行人检测算法[J]. 计算机工程, 2024, 50(7): 342-351.
[8]	程腾腾, 姚春龙, 于晓强, 李旭, 王庆丰. 基于多头注意力机制融合常识知识的共情对话生成[J]. 计算机工程, 2024, 50(6): 94-101.
[9]	更藏措毛, 黄鹤鸣, 杨毅杰. 融合多尺度特征与上下文信息的语音增强方法[J]. 计算机工程, 2024, 50(6): 138-147.
[10]	贺姗, 蔺素珍, 王彦博, 李大威. 基于特征融合的多波段图像描述生成方法[J]. 计算机工程, 2024, 50(6): 236-244.
[11]	杨硕, 王一丁. 基于改进薄板样条运动模型的人脸动画算法[J]. 计算机工程, 2024, 50(6): 255-265.
[12]	陈晓玉, 沈晨, 沈阅, 孔德明. 基于改进SwiftNet的堆场图像实时分割网络[J]. 计算机工程, 2024, 50(6): 296-303.
[13]	游奔, 李晓红, 姚锦, 冯绍杰. 基于多粒度图与注意力机制的半监督短文本分类[J]. 计算机工程, 2024, 50(5): 83-90.
[14]	曹渝昆, 程宇, 何祯奕, 徐康乐, 颜家洛, 李云峰. 文档上下文异构表示的句子级关系抽取方法[J]. 计算机工程, 2024, 50(5): 111-119.
[15]	代巍, 王丰羽, 冀常鹏. 基于情感增强与双图卷积网络的方面级情感分析[J]. 计算机工程, 2024, 50(5): 120-127.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

一种注意力增强的自然语言推理模型

An Attention-enchanced Natural Language Reasoning Model

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价