Chinese Machine Reading Comprehension Based on Hybrid Attention Mechanism

doi:10.19678/j.issn.1000-3428.0062206

Abstract

Abstract: The pre-training language model performs well in the field of machine reading comprehension.Compared with English machine reading comprehension, the reading comprehension model based on the pre-training language model performs slightly worse in processing Chinese text and can only learn the shallow semantic matching of the text.To improve the ability of the model to understand Chinese text, this paper proposes a Chinese machine reading comprehension model based on hybrid attention mechanism.The model uses the pre-training model to obtain the sequence representation in the coding layer and further deepens the context interaction through BiLSTM processing.Then, this is processed by a hybrid attention layer comprising two variants of self-attention mechanism, which aims to learn the deep semantic representation, to deepen the understanding of the text semantic information.Further, the fusion layer combines multiple fusion mechanisms to obtain the multi-level representation, making the output sequence carry more rich information.Finally, after double BiLSTM processing, input output layer to get the answer position.The experimental results on CMRC2018 dataset show that the EM and F1 values of this model are increased by 2.05 and 0.465 percentage points, respectively, compared with those of the baseline model.This enables to learn the deep semantic information of the text and effectively improve the pre-trained language model.

Key words: Chinese machine reading comprehension, attention mechanism, fusion mechanism, pre-training model, RoBERTa model

摘要： 预训练语言模型在机器阅读理解领域具有较好表现，但相比于英文机器阅读理解，基于预训练语言模型的阅读理解模型在处理中文文本时表现较差，只能学习文本的浅层语义匹配信息。为了提高模型对中文文本的理解能力，提出一种基于混合注意力机制的阅读理解模型。该模型在编码层使用预训练模型得到序列表示，并经过BiLSTM处理进一步加深上下文交互，再通过由两种变体自注意力组成的混合注意力层处理，旨在学习深层语义表示，以加深对文本语义信息的理解，而融合层结合多重融合机制获取多层次的表示，使得输出的序列携带更加丰富的信息，最终使用双层BiLSTM处理输入输出层得到答案位置。在CMRC2018数据集上的实验结果表明，与复现的基线模型相比，该模型的EM值和F1值分别提升了2.05和0.465个百分点，能够学习到文本的深层语义信息，有效改进预训练语言模型。

关键词: 中文机器阅读理解, 注意力机制, 融合机制, 预训练模型, RoBERTa模型

CLC Number:

TP391

LIU Gaojun, LI Yaxin, DUAN Jianyong. Chinese Machine Reading Comprehension Based on Hybrid Attention Mechanism[J]. Computer Engineering, 2022, 48(10): 67-72,80.

刘高军, 李亚欣, 段建勇. 基于混合注意力机制的中文机器阅读理解[J]. 计算机工程, 2022, 48(10): 67-72,80.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0062206

https://www.ecice06.com/EN/Y2022/V48/I10/67

Figures/Tables 9

References

[1] CHEN D Q.Neural reading comprehension and beyond[D].PaloAlto, USA:Stanford University, 2018.
[2] DEVLIN J, CHANG M W, LEE K, et al.BERT:pre-training of deep bidirectional transformers for language understanding[EB/OL].[2021-06-20].https://arxiv.org/abs/1810.04805.
[3] LAN Z Z, CHEN M D, GOODMAN S, et al.ALBERT:a lite BERT for self-supervised learning of language representations[EB/OL].[2021-06-20].https://arxiv.org/abs/1909.11942.
[4] LIU Y H, OTT M, GOYAL N, et al.RoBERTa:a robustly optimized BERT pretraining approach[EB/OL].[2021-06-20].https://arxiv.org/abs/1907.11692.
[5] BAHDANAU D, CHO K, BENGIO Y.Neural machine translation by jointly learning to align and translate[EB/OL].[2021-06-20].https://arxiv.org/abs/1409.0473.
[6] HERMANN K M, KOČISKÝ T, GREFENSTETTE E, et al.Teaching machines to read and comprehend[EB/OL].[2021-06-20].https://arxiv.org/abs/1506.03340.
[7] KADLEC R, SCHMID M, BAJGAR O, et al.Text understanding with the attention sum reader network[EB/OL].[2021-06-20].https://arxiv.org/abs/1603.01547.
[8] CHEN D Q, BOLTON J, MANNING C D.A thorough examination of the CNN/daily mail reading comprehension task[EB/OL].[2021-06-20].https://arxiv.org/abs/1606.02858.
[9] SEO M, KEMBHAVI A, FARHADI A, et al.Bidirectional attention flow for machine comprehension[EB/OL].[2021-06-20].https://arxiv.org/abs/1611.01603.
[10] CHEN D Q, FISCH A, WESTON J, et al.Reading Wikipedia to answer open-domain questions[EB/OL].[2021-06-20].https://arxiv.org/abs/1704.00051v2.
[11] WANG W H, YANG N, WEI F R, et al.Gated self-matching networks for reading comprehension and question answering[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.Washington D.C., USA:IEEE Press, 2017:189-198.
[12] HUANG H Y, ZHU C G, SHEN Y L, et al.FusionNet:fusing via fully-aware attention with application to machine comprehension[EB/OL].[2021-06-20].https://arxiv.org/abs/1711.07341.
[13] VASWANI A, SHAZEER N, PARMAR N, et al.Attention is all you need[C]//Proceedings of NIPSʼ17.Cambridge, USA:MIT Press, 2017:5998-6008.
[14] CUI Y M, LIU T, CHEN Z, et al.Consensus attention-based neural networks for Chinese reading comprehension[EB/OL].[2021-06-20].https://arxiv.org/abs/1607.02250.
[15] CUI Y M, LIU T, CHE W X, et al.A span-extraction dataset for Chinese machine reading comprehension[EB/OL].[2021-06-20].https://arxiv.org/abs/1810.07366.
[16] CUI Y M, LIU T, YANG Z Q, et al.A sentence cloze dataset for Chinese machine reading comprehension[EB/OL].[2021-06-20].https://arxiv.org/abs/2004.03116.
[17] HE W, LIU K, LIU J, et al.Dureader:a chinese machine reading comprehension dataset from real-world applications[C]//Proceedings of Workshop on Machine Reading for Question Answering.Washington D.C., USA:IEEE Press, 2018:37-46.
[18] 徐丽丽, 李茹, 李月香, 等.面向机器阅读理解的语句填补答案选择方法[J].计算机工程, 2018, 44(7):183-187, 192. XU L L, LI R, LI Y X, et al.Answer selection method of sentence filling for machine reading comprehension[J].Computer Engineering, 2018, 44(7):183-187, 192.(in Chinese)
[19] SHAO C C, LIU T, LAI Y T, et al.DRCD:a Chinese machine reading comprehension dataset[EB/OL].[2021-06-20].https://arxiv.org/abs/1806.00920.
[20] CUI Y M, CHE W X, LIU T, et al.Pre-training with whole word masking for Chinese BERT[EB/OL].[2021-06-20].https://arxiv.org/abs/1906.08101.
[21] TAY Y, BAHRI D, METZLER D, et al.Synthesizer:rethinking self-attention in transformer models[EB/OL].[2021-06-20].https://arxiv.org/abs/2005.00743.
[22] CUI Y M, CHE W X, LIU T, et al.Revisiting pre-trained models for Chinese natural language processing[EB/OL].[2021-06-20].https://arxiv.org/abs/2004.13922.

Please choose a citation manager

Content to export