融合ELECTRA和文本局部信息的中文语法错误检测方法

doi:10.19678/j.issn.1000-3428.0064014

摘要/Abstract

摘要： 语法错误检测是自然语言处理领域的一项基本任务，其目标是自动识别文本中存在的错别字、语法及语序错误等。与其他语言相比，中文语法灵活多变且缺乏时态、语态等标志性信息，因此，文本的局部信息对于中文语法错误检测具有重要作用。传统的机器学习方法难以检测文本中存在的语法错误，而现有深度学习方法在纠错过程中不能充分利用文本的局部信息，导致语法错误检测效果不佳。建立一种融合ELECTRA和文本局部信息的中文语法错误检测模型ELECTRA-GCNN-CRF。将语法错误检测视为序列标注任务，使用ELECTRA预训练语言模型对文本进行表征。采用卷积神经网络提取文本的局部位置和语义信息，并引入残差门控机制，降低无效信息带来的影响。通过CRF模型学习标签间的内在关联关系，输出符合标注规则的语法错误标签序列。在NLPTEA中文语法错误检测数据集上的实验结果表明，ELECTRA-GCNN-CRF在检测层、识别层和定位层上的F1值较对比基线模型分别平均提高了0.94、3.74和5.03个百分点，该模型能够有效提升语法错误检测效果。

关键词: ELECTRA预训练语言模型, 局部信息, 中文语法错误检测, 卷积神经网络, 残差门控机制

Abstract: Grammar error detection is a basic task in natural language processing.The task aims to automatically identify typos, grammar, and word order errors in text.Compared with other languages, Chinese grammar is flexible and lacks symbolic information such as tense and voice.Therefore, the local information of the text plays an important role in Chinese Grammar Error Detection(CGED).Conventional machine learning methods are difficult to detect grammatical errors in a text, whereas the existing deep learning methods cannot utilize the local information of the text during error correction fully and effectively, resulting in poor grammatical error detection effect.To solve this problem, this study proposes a CGED model, ELECTRA-GCNN-CRF, integrating an ELECTRA and the local information of the text.Grammar error detection is regarded as a sequence annotation task.First, the text is represented by an ELECTRA pre-training language model.Second, a Convolution Neural Network(CNN) is used to extract the local position and semantic information of the text and the residual and gating mechanisms are introduced to reduce the impact of invalid information.Finally, the internal relationship between tags is learned through a CRF model, and the grammar error tag sequence conforming to the labeling rules is output.The model proposed in this study is tested on the Chinese grammatical error evaluation dataset of NLPTEA.The F1 values of detection-, identification-, and position-level increased by 0.94, 3.74, and 5.03 percentage points, respectively, compared with the baseline model, which improves the effect of grammatical error detection.

Key words: ELECTRA pre-training language model, local information, Chinese Grammar Error Detection(CGED), Convolution Neural Network(CNN), residual gated mechanism

中图分类号:

TP391

陈柏霖, 王天极, 任丽娜, 黄瑞章. 融合ELECTRA和文本局部信息的中文语法错误检测方法[J]. 计算机工程, 2023, 49(3): 304-311.

CHEN Bailin, WANG Tianji, REN Lina, HUANG Ruizhang. Method for Chinese Grammar Error Detection Integrating ELECTRA and Text Local Information[J]. Computer Engineering, 2023, 49(3): 304-311.

http://www.ecice06.com/CN/Y2023/V49/I3/304

图/表 10

20230314190921

20230314190924

20230314190927

20230314190931

20230314190934

20230314190938

20230314190941

20230314190944

20230314190947

20230314190950

参考文献

[1] 赵国红.中文语法纠错方法的研究综述[J].现代计算机, 2021, 27(28):65-69. ZHAO G H.A survey of researches on Chinese grammar error correction methods[J].Modern Computer, 2021, 27(28):65-69.(in Chinese)
[2] JI J S, WANG Q L, TOUTANOVA K, et al.A nested attention neural hybrid model for grammatical error correction[EB/OL].[2022-01-05].https://arxiv.org/pdf/1707.02026.pdf.
[3] CHOLLAMPATT S, NG H T.A multilayer convolutional encoder-decoder neural network for grammatical error correction[EB/OL].[2022-01-05].https://arxiv.org/pdf/1801.08831.pdf.
[4] FU K, HUANG J, DUAN Y T.Youdao's winning solution to the NLPCC-2018 task 2 challenge:a neural machine translation approach to Chinese grammatical error correction[EB/OL].[2022-01-05].http://tcci.ccf.org.cn/conference/2018/papers/EV39.pdf.
[5] ZHAO W, WANG L, SHEN K W, et al.Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data[EB/OL].[2022-01-05].https://arxiv.org/abs/1903.00138.
[6] CHENG X Y, XU W D, CHEN K L, et al.SpellGCN:incorporating phonological and visual similarities into language models for Chinese spelling check[EB/OL].[2022-01-05].https://arxiv.org/abs/2004.14166.
[7] LIU S L, YANG T, YUE T C, et al.PLOME:pre-training with misspelled knowledge for Chinese spelling correction[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing.[S.l.]:Association for Computational Linguistics, 2021:2991-3000.
[8] HUANG L, LI J J, JIANG W W, et al.PHMOSpell:phonological and morphological knowledge guided Chinese spelling check[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing.[S.l.]:Association for Computational Linguistics, 2021:5958-5967.
[9] ZHANG R Q, PANG C, ZHANG C Q, et al.Correcting Chinese spelling errors with phonetic pre-training[EB/OL].[2022-01-05].https://aclanthology.org/2021.findings-acl.198.pdf.
[10] SUN Z J, LI X Y, SUN X F, et al.ChineseBERT:Chinese pretraining enhanced by glyph and pinyin information[EB/OL].[2022-01-05].https://arxiv.org/abs/2106.16038.
[11] HONG Y Z, YU X G, HE N, et al.FASPell:a fast, adaptable, simple, powerful Chinese spell checker based on DAE-decoder paradigm[C]//Proceedings of the 5th Workshop on Noisy User-Generated Text.[S.l.]:Association for Computational Linguistics, 2019:160-169.
[12] GUO Z, NI Y, WANG K Q, et al.Global attention decoder for Chinese spelling error correction[EB/OL].[2022-01-05].https://aclanthology.org/2021.findings-acl.122.pdf.
[13] WANG B X, CHE W X, WU D Y, et al.Dynamic connected networks for Chinese spelling check[EB/OL].[2022-01-05].https://aclanthology.org/2021.findings-acl.216.pdf.
[14] 谢海华, 陈志优, 程静, 等.基于数据增强和多任务特征学习的中文语法错误检测方法[EB/OL].[2022-01-05].https://www.xueshufan.com/publication/3102634486. XIE H H, CHEN Z Y, CHENG J, et al.Chinese grammar error detection based on data enhancement and multi-task feature learning[EB/OL].[2022-01-05].https://www.xueshufan.com/publication/3102634486.(in Chinese)
[15] 王辰成, 杨麟儿, 王莹莹, 等.基于Transformer增强架构的中文语法纠错方法[J].中文信息学报, 2020, 34(6):106-114. WANG C C, YANG L E, WANG Y Y, et al.Chinese grammatical error correction method based on Transformer enhanced architecture[J].Journal of Chinese Information Processing, 2020, 34(6):106-114.(in Chinese)
[16] FU R J, PEI Z Q, GONG J F, et al.Chinese grammatical error diagnosis using statistical and prior knowledge driven features with probabilistic ensemble enhancement[C]//Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications.[S.l.]:Association for Computational Linguistics, 2018:52-59.
[17] RAO G, YANG E, ZHANG B.Overview of NLPTEA-2020 shared task for Chinese grammatical error diagnosis[C]//Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications.[S.l.]:Association for Computational Linguistics, 2020:25-35.
[18] WANG S, WANG B, GONG J, et al.Combining ResNet and Transformer for Chinese grammatical error diagnosis[C]//Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications.[S.l.]:Association for Computational Linguistics, 2020:36-43.
[19] CAO Y, HE L, RIDLEY R, et al.Integrating BERT and score-based feature gates for Chinese grammatical error diagnosis[C]//Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications.[S.l.]:Association for Computational Linguistics, 2020:49-56.
[20] LUO Y, BAO Z, LI C, et al.Chinese grammatical error diagnosis with graph convolution network and multi-task learning[C]//Proceedings of the 6th Workshop on Natural Language Processing Techniques for Educational Applications.[S.l.]:Association for Computational Linguistics, 2020:44-48.
[21] CLARK K, LUONG M T, LE Q V, et al.ELECTRA:pre-training text encoders as discriminators rather than generators[EB/OL].[2022-01-05].https://arxiv.org/abs/2003.10555.
[22] SALIMANS T, GOODFELLOW I, ZAREMBA W, et al.Improved techniques for training GANs[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems.New York, USA:ACM Press, 2016:2234-2242.
[23] DAUPHIN Y N, FAN A, AULI M, et al.Language modeling with gated convolutional networks[C]//Proceedings of the 34th International Conference on Machine Learning.New York, USA:ACM Press, 2017:933-941.
[24] CUI Y M, CHE W X, LIU T, et al.Pre-training with whole word masking for Chinese BERT[J].IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29:3504-3514.
[25] DEVLIN J, CHANG M W, LEE K, et al.BERT:pre-training of deep bidirectional transformers for language understanding[EB/OL].[2022-01-05].https://arxiv.org/abs/1810.04805.

选择文件类型/文献管理软件名称

选择包含的内容