基于混合注意力机制的中文机器阅读理解

doi:10.19678/j.issn.1000-3428.0062206

计算机工程 ›› 2022, Vol. 48 ›› Issue (10): 67-72,80. doi: 10.19678/j.issn.1000-3428.0062206

基于混合注意力机制的中文机器阅读理解

刘高军^1,2, 李亚欣^1,2, 段建勇^1,2

1. 北方工业大学信息学院, 北京 100144;
2. 北方工业大学CNONIX国家标准应用与推广实验室, 北京 100144

收稿日期:2021-07-29 修回日期:2021-11-03 发布日期:2021-11-15
作者简介:刘高军(1962—),男,教授,主研方向为数据处理、软件服务;李亚欣,硕士研究生;段建勇,教授。
基金资助:
国家自然科学基金（61972003，61672040）。

Chinese Machine Reading Comprehension Based on Hybrid Attention Mechanism

LIU Gaojun^1,2, LI Yaxin^1,2, DUAN Jianyong^1,2

1. School of Information, North China University of Technology, Beijing 100144, China;
2. CNONIX National Standard Application and Promotion Laboratory, North China University of Technology, Beijing 100144, China

Received:2021-07-29 Revised:2021-11-03 Published:2021-11-15

摘要/Abstract

摘要： 预训练语言模型在机器阅读理解领域具有较好表现，但相比于英文机器阅读理解，基于预训练语言模型的阅读理解模型在处理中文文本时表现较差，只能学习文本的浅层语义匹配信息。为了提高模型对中文文本的理解能力，提出一种基于混合注意力机制的阅读理解模型。该模型在编码层使用预训练模型得到序列表示，并经过BiLSTM处理进一步加深上下文交互，再通过由两种变体自注意力组成的混合注意力层处理，旨在学习深层语义表示，以加深对文本语义信息的理解，而融合层结合多重融合机制获取多层次的表示，使得输出的序列携带更加丰富的信息，最终使用双层BiLSTM处理输入输出层得到答案位置。在CMRC2018数据集上的实验结果表明，与复现的基线模型相比，该模型的EM值和F1值分别提升了2.05和0.465个百分点，能够学习到文本的深层语义信息，有效改进预训练语言模型。

关键词: 中文机器阅读理解, 注意力机制, 融合机制, 预训练模型, RoBERTa模型

Abstract: The pre-training language model performs well in the field of machine reading comprehension.Compared with English machine reading comprehension, the reading comprehension model based on the pre-training language model performs slightly worse in processing Chinese text and can only learn the shallow semantic matching of the text.To improve the ability of the model to understand Chinese text, this paper proposes a Chinese machine reading comprehension model based on hybrid attention mechanism.The model uses the pre-training model to obtain the sequence representation in the coding layer and further deepens the context interaction through BiLSTM processing.Then, this is processed by a hybrid attention layer comprising two variants of self-attention mechanism, which aims to learn the deep semantic representation, to deepen the understanding of the text semantic information.Further, the fusion layer combines multiple fusion mechanisms to obtain the multi-level representation, making the output sequence carry more rich information.Finally, after double BiLSTM processing, input output layer to get the answer position.The experimental results on CMRC2018 dataset show that the EM and F1 values of this model are increased by 2.05 and 0.465 percentage points, respectively, compared with those of the baseline model.This enables to learn the deep semantic information of the text and effectively improve the pre-trained language model.

Key words: Chinese machine reading comprehension, attention mechanism, fusion mechanism, pre-training model, RoBERTa model

中图分类号:

TP391

刘高军, 李亚欣, 段建勇. 基于混合注意力机制的中文机器阅读理解[J]. 计算机工程, 2022, 48(10): 67-72,80.

LIU Gaojun, LI Yaxin, DUAN Jianyong. Chinese Machine Reading Comprehension Based on Hybrid Attention Mechanism[J]. Computer Engineering, 2022, 48(10): 67-72,80.

http://www.ecice06.com/CN/Y2022/V48/I10/67

图/表 9

参考文献

[1] CHEN D Q.Neural reading comprehension and beyond[D].PaloAlto, USA:Stanford University, 2018.
[2] DEVLIN J, CHANG M W, LEE K, et al.BERT:pre-training of deep bidirectional transformers for language understanding[EB/OL].[2021-06-20].https://arxiv.org/abs/1810.04805.
[3] LAN Z Z, CHEN M D, GOODMAN S, et al.ALBERT:a lite BERT for self-supervised learning of language representations[EB/OL].[2021-06-20].https://arxiv.org/abs/1909.11942.
[4] LIU Y H, OTT M, GOYAL N, et al.RoBERTa:a robustly optimized BERT pretraining approach[EB/OL].[2021-06-20].https://arxiv.org/abs/1907.11692.
[5] BAHDANAU D, CHO K, BENGIO Y.Neural machine translation by jointly learning to align and translate[EB/OL].[2021-06-20].https://arxiv.org/abs/1409.0473.
[6] HERMANN K M, KOČISKÝ T, GREFENSTETTE E, et al.Teaching machines to read and comprehend[EB/OL].[2021-06-20].https://arxiv.org/abs/1506.03340.
[7] KADLEC R, SCHMID M, BAJGAR O, et al.Text understanding with the attention sum reader network[EB/OL].[2021-06-20].https://arxiv.org/abs/1603.01547.
[8] CHEN D Q, BOLTON J, MANNING C D.A thorough examination of the CNN/daily mail reading comprehension task[EB/OL].[2021-06-20].https://arxiv.org/abs/1606.02858.
[9] SEO M, KEMBHAVI A, FARHADI A, et al.Bidirectional attention flow for machine comprehension[EB/OL].[2021-06-20].https://arxiv.org/abs/1611.01603.
[10] CHEN D Q, FISCH A, WESTON J, et al.Reading Wikipedia to answer open-domain questions[EB/OL].[2021-06-20].https://arxiv.org/abs/1704.00051v2.
[11] WANG W H, YANG N, WEI F R, et al.Gated self-matching networks for reading comprehension and question answering[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.Washington D.C., USA:IEEE Press, 2017:189-198.
[12] HUANG H Y, ZHU C G, SHEN Y L, et al.FusionNet:fusing via fully-aware attention with application to machine comprehension[EB/OL].[2021-06-20].https://arxiv.org/abs/1711.07341.
[13] VASWANI A, SHAZEER N, PARMAR N, et al.Attention is all you need[C]//Proceedings of NIPSʼ17.Cambridge, USA:MIT Press, 2017:5998-6008.
[14] CUI Y M, LIU T, CHEN Z, et al.Consensus attention-based neural networks for Chinese reading comprehension[EB/OL].[2021-06-20].https://arxiv.org/abs/1607.02250.
[15] CUI Y M, LIU T, CHE W X, et al.A span-extraction dataset for Chinese machine reading comprehension[EB/OL].[2021-06-20].https://arxiv.org/abs/1810.07366.
[16] CUI Y M, LIU T, YANG Z Q, et al.A sentence cloze dataset for Chinese machine reading comprehension[EB/OL].[2021-06-20].https://arxiv.org/abs/2004.03116.
[17] HE W, LIU K, LIU J, et al.Dureader:a chinese machine reading comprehension dataset from real-world applications[C]//Proceedings of Workshop on Machine Reading for Question Answering.Washington D.C., USA:IEEE Press, 2018:37-46.
[18] 徐丽丽, 李茹, 李月香, 等.面向机器阅读理解的语句填补答案选择方法[J].计算机工程, 2018, 44(7):183-187, 192. XU L L, LI R, LI Y X, et al.Answer selection method of sentence filling for machine reading comprehension[J].Computer Engineering, 2018, 44(7):183-187, 192.(in Chinese)
[19] SHAO C C, LIU T, LAI Y T, et al.DRCD:a Chinese machine reading comprehension dataset[EB/OL].[2021-06-20].https://arxiv.org/abs/1806.00920.
[20] CUI Y M, CHE W X, LIU T, et al.Pre-training with whole word masking for Chinese BERT[EB/OL].[2021-06-20].https://arxiv.org/abs/1906.08101.
[21] TAY Y, BAHRI D, METZLER D, et al.Synthesizer:rethinking self-attention in transformer models[EB/OL].[2021-06-20].https://arxiv.org/abs/2005.00743.
[22] CUI Y M, CHE W X, LIU T, et al.Revisiting pre-trained models for Chinese natural language processing[EB/OL].[2021-06-20].https://arxiv.org/abs/2004.13922.

选择文件类型/文献管理软件名称

选择包含的内容

基于混合注意力机制的中文机器阅读理解

Chinese Machine Reading Comprehension Based on Hybrid Attention Mechanism

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	丰芳宇, 罗晓曙, 蒙志明, 王广宇. 基于抗混叠残差注意力网络的人脸表情识别[J]. 计算机工程, 2023, 49(8): 190-198.
[2]	王书朋, 何引弟. 融合特征注意力机制的非均匀光照图像增强算法[J]. 计算机工程, 2023, 49(8): 232-239.
[3]	刘昊鑫, 董超, 勾智楠, 高凯. 融合混合表征的小样本关系抽取方法[J]. 计算机工程, 2023, 49(8): 63-68.
[4]	杨长沛, 廖列法. 基于门控空洞卷积特征融合的中文命名实体识别[J]. 计算机工程, 2023, 49(8): 85-95.
[5]	刘俊豪, 王美林, 谢兴, 宋烨兴, 许莉花. 基于改进YOLOv5的皮革瑕疵检测算法[J]. 计算机工程, 2023, 49(8): 240-249.
[6]	马娜, 温廷新, 贾旭, 李晓会. 复杂光照条件下自适应的车脸重识别模型[J]. 计算机工程, 2023, 49(8): 275-282, 290.
[7]	陈露萌, 曹彦彦, 黄民, 谢鑫钢. 基于改进YOLOv5的火焰检测方法[J]. 计算机工程, 2023, 49(8): 291-301, 309.
[8]	李强龙, 周新文, 位梦恩, 甘阳洲. 基于条形池化和注意力机制的街道场景红外目标检测算法[J]. 计算机工程, 2023, 49(8): 310-320.
[9]	张家熔, 苑津莎, 许珈宁, 罗志宏. 基于多元信息嵌入与协同神经网络的力学实体识别算法[J]. 计算机工程, 2023, 49(7): 125-134.
[10]	白明昌. 基于折叠路径聚合的属性网络节点嵌入方法[J]. 计算机工程, 2023, 49(7): 76-84.
[11]	费蓉, 马梦阳, 张晓, 黑新宏, 徐庆征, 邱原. 基于轨迹预测与冲突检测的自动驾驶碰撞检测模型[J]. 计算机工程, 2023, 49(7): 10-20.
[12]	刘豪, 吴红兰, 房宇轩. 结合全局上下文信息的高效人体姿态估计[J]. 计算机工程, 2023, 49(7): 102-109.
[13]	吴珊, 周凤. 基于改进SSD算法的小目标检测[J]. 计算机工程, 2023, 49(7): 179-188.
[14]	齐咏生, 杜晓旭, 朱俊峰, 高胜利, 刘利强. 基于增强型轻量深度网络的牧区牲畜高效检测[J]. 计算机工程, 2023, 49(7): 278-287.
[15]	陈明, 刘蓉, 张晔. 基于多重注意力机制的中文医疗实体识别[J]. 计算机工程, 2023, 49(6): 314-320.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于混合注意力机制的中文机器阅读理解

Chinese Machine Reading Comprehension Based on Hybrid Attention Mechanism

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献

相关文章 15

编辑推荐

Metrics

本文评价