基于CNN‐Head Transformer编码器的中文命名实体识别

doi:10.19678/j.issn.1000-3428.0062525

计算机工程 ›› 2022, Vol. 48 ›› Issue (10): 73-80. doi: 10.19678/j.issn.1000-3428.0062525

基于CNN‐Head Transformer编码器的中文命名实体识别

史占堂^1,2,3, 马玉鹏^1,2,3, 赵凡^1,2,3, 马博^1,2,3

1. 中国科学院新疆理化技术研究所, 乌鲁木齐 830011;
2. 中国科学院大学, 北京 100049;
3. 新疆民族语音语言信息处理实验室, 乌鲁木齐 830011

收稿日期:2021-08-28 修回日期:2021-10-22 发布日期:2021-11-16
作者简介:史占堂(1995—),男,硕士研究生,主研方向为自然语言处理;马玉鹏、赵凡,研究员、博士;马博,副研究员、博士。
基金资助:
国家重点研发计划（2018YFC0825300，2018YFC0823002）；新疆维吾尔自治区重大专项（2020A03004-4）；“西部之光”人才培养计划（2018-XBQNXZ-A-003）；中国科学院青年创新促进会基金（科发人函字［2019］26号）。

Chinese Named Entity Recognition Based on CNN-Head Transformer Encoder

SHI Zhantang^1,2,3, MA Yupeng^1,2,3, ZHAO Fan^1,2,3, MA Bo^1,2,3

1. Xinjiang Technical Institute of Physics and Chemistry, Chinese Academy of Sciences, Urumqi 830011, China;
2. University of Chinese Academy of Sciences, Beijing 100049, China;
3. Xinjiang Laboratory of Minority Speech and Language Information Processing, Urumqi 830011, China

Received:2021-08-28 Revised:2021-10-22 Published:2021-11-16

摘要/Abstract

摘要： 基于多头自注意力机制的Transformer作为主流特征提取器在多种自然语言处理任务中取得了优异表现，但应用于命名实体识别任务时存在一字多词、增加额外存储与词典匹配时间等问题。提出一种CNN-Head Transformer编码器（CHTE）模型，在未使用外部词典和分词工具的基础上，通过自注意力机制捕获全局语义信息，利用不同窗口大小的CNN获取Transformer中6个注意力头的Value向量，使CHTE模型在保留全局语义信息的同时增强局部特征和潜在词信息表示，并且应用自适应的门控残差连接融合当前层和子层特征，提升了Transformer在命名实体识别领域的性能表现。在Weibo和Resume数据集上的实验结果表明，CHTE模型的F1值相比于融合词典信息的Lattice LSTM和FLAT模型分别提升了3.77、2.24和1.30、0.31个百分点，具有更高的中文命名实体识别准确性。

关键词: 命名实体识别, 自注意力机制, Transformer编码器, 卷积神经网络, 残差连接

Abstract: Transformers based on the multihead self-attention mechanism as a mainstream feature extractor have achieved excellent performance in various Natural Language Processing(NLP) tasks.However, problems such as one character belonging to multiple words, additional storage, and dictionary matching time arise when those transformers are applied to Named Entity Recognition(NER) tasks.This study proposes a Convolutional Neural Network(CNN)-Head Transformer Encoder(CHTE) model to capture global semantic information via a self-attention mechanism without using external lexicon and word segmentation tools.Additionally, a CNN with different window sizes is used to obtain the value vectors of six attention heads in the transformer.It preserves global semantic information while enhancing local features and latent word information representation, as well as applies adaptive gated residual connection to fuse current layer and sublayer features to improve the performance of the transformer in NER tasks.Experimental results show that the F1 values of the CHTE model on the Weibo dataset are 3.77 and 2.24 percentage points higher than those of the Lattice Long Short-Term Memory(Lattice LSTM) and FLAT-lattice Transformer(FLAT) models fused with dictionary information, respectively;the F1 values of the CHTE model on the Resume dataset improved by 1.30 and 0.31 percentage points than those of the Lattice LSTM and FLAT models, respectively;and the CHTE model demonstrates higher recognition accuracy for Chinese named entities than Lattice LSTM and FLAT models.

Key words: Named Entity Recognition(NER), self-attention mechanism, Transformer encoder, Convolutional Neural Network(CNN), residual connection

中图分类号:

TP391

史占堂, 马玉鹏, 赵凡, 马博. 基于CNN‐Head Transformer编码器的中文命名实体识别[J]. 计算机工程, 2022, 48(10): 73-80.

SHI Zhantang, MA Yupeng, ZHAO Fan, MA Bo. Chinese Named Entity Recognition Based on CNN-Head Transformer Encoder[J]. Computer Engineering, 2022, 48(10): 73-80.

http://www.ecice06.com/CN/Y2022/V48/I10/73

图/表 9

参考文献

[1] WANG L L, CAO Z, DE MELO G, et al.Relation classification via multi-level attention CNNs[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.Stroudsburg, USA:Association for Computational Linguistics, 2016:1298-1307.
[2] LIU X, LUO Z C, HUANG H Y.Jointly multiple events extraction via attention-based graph information aggregation[EB/OL].[2021-07-11].https://arxiv.org/abs/1809.09078.
[3] KOLITSAS N, GANEA O E, HOFMANN T.End-to-end neural entity linking[EB/OL].[2021-07-11].https://arxiv.org/abs/1808.07699.
[4] GERS F A, SCHMIDHUBER J, CUMMINS F.Learning to forget:continual prediction with LSTM[J].Neural Computation, 2000, 12(10):2451-2471.
[5] ZHANG Y, YANG J.Chinese NER using lattice LSTM[EB/OL].[2021-07-11].https://arxiv.org/abs/1805.02023.
[6] VASWANI A, SHAZEER N, PARMAR N, et al.Attention is all you need[EB/OL].[2021-07-11].https://arxiv.org/abs/1706.03762.
[7] YAN H, DENG B C, LI X N, et al.TENER:adapting Transformer encoder for named entity recognition[EB/OL].[2021-07-11].https://arxiv.org/abs/1911.04474.
[8] LECUN Y, BOSER B, DENKER J S, et al.Backpropagation applied to handwritten Zip code recognition[J].Neural Computation, 1989, 1(4):541-551.
[9] HUANG Z H, XU W, YU K.Bidirectional LSTM-CRF models for sequence tagging[EB/OL].[2021-07-11].https://arxiv.org/abs/1508.01991.
[10] LAFFERTY J, MCCALLUM A, PEREIRA F.Conditional random fields:probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the 18th International Conference on Machine Learning.New York, USA:ACM Press, 2001:282-289.
[11] PENG N Y, DREDZE M.Named entity recognition for Chinese social media with jointly trained embeddings[C]//Proceedings of 2015 Conference on Empirical Methods in Natural Language Processing.Stroudsburg, USA:Association for Computational Linguistics, 2015:548-554.
[12] MA X Z, HOVY E.End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF[EB/OL].[2021-07-11].https://arxiv.org/abs/1603.01354.
[13] STRUBELL E, VERGA P, BELANGER D, et al.Fast and accurate entity recognition with iterated dilated convolutions[C]//Proceedings of 2017 Conference on Empirical Methods in Natural Language Processing.Stroudsburg, USA:Association for Computational Linguistics, 2017:1-11.
[14] LIU Y J, ZHANG Y, CHE W X, et al.Domain adaptation for CRF-based Chinese word segmentation using free annotations[C]//Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing.Stroudsburg, USA:Association for Computational Linguistics, 2014:864-874.
[15] CAO P F, CHEN Y B, LIU K, et al.Adversarial transfer learning for Chinese named entity recognition with self-attention mechanism[C]//Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing.Stroudsburg, USA:Association for Computational Linguistics, 2018:182-192.
[16] GUI T, MA R T, ZHANG Q, et al.CNN-based Chinese NER with lexicon rethinking[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence.New York, USA:ACM Press, 2019:4982-4988.
[17] MENG Y X, WU W, WANG F, et al.Glyce:glyph-vectors for Chinese character representations[EB/OL].[2021-07-11].https://arxiv.org/abs/1901.10125.
[18] XUAN Z Y, BAO R, JIANG S Y.FGN:fusion glyph network for Chinese named entity recognition[M].Berlin, Germany:Springer, 2021.
[19] 张栋, 王铭涛, 陈文亮.结合五笔字形与上下文相关字向量的命名实体识别[J].计算机工程, 2021, 47(3):94-101. ZHANG D, WANG M T, CHEN W L.Named entity recognition combining Wubi glyphs with contextualized character embeddings[J].Computer Engineering, 2021, 47(3):94-101.(in Chinese)
[20] 司逸晨, 管有庆.基于Transformer编码器的中文命名实体识别模型[J].计算机工程, 2022, 48(7):66-72. SI Y C, GUAN Y Q.Chinese named entity recognition model based on Transformer encoder[J].Computer Engineering, 2022, 48(7):66-72.(in Chinese)
[21] XUE M G, YU B W, LIU T W, et al.Porous Lattice Transformer encoder for Chinese NER[C]//Proceedings of the 28th International Conference on Computational Linguistics.Stroudsburg, USA:International Committee on Computational Linguistics, 2020:3831-3841.
[22] LI X N, YAN H, QIU X P, et al.FLAT:Chinese NER using FLAT-lattice Transformer[EB/OL].[2021-07-11].https://arxiv.org/abs/2004.11795.
[23] HE H F, SUN X.F-score driven max margin neural network for named entity recognition in Chinese social media[EB/OL].[2021-07-11].https://arxiv.org/abs/1611.04234.
[24] ZHU Y Y, WANG G X, KARLSSON B F.CAN-NER:convolutional attention network for Chinese named entity recognition[EB/OL].[2021-07-11].https://arxiv.org/abs/1904.02141.
[25] CUI Y M, CHE W X, LIU T, et al.Pre-training with whole word masking for Chinese BERT[EB/OL].[2021-07-11].https://arxiv.org/abs/1906.08101.
[26] DEVLIN J, CHANG M W, LEE K, et al.BERT:pre-training of deep bidirectional Transformers for language understanding[EB/OL].[2021-07-11].https://arxiv.org/abs/1810.04805.

选择文件类型/文献管理软件名称

选择包含的内容

基于CNN‐Head Transformer编码器的中文命名实体识别

Chinese Named Entity Recognition Based on CNN-Head Transformer Encoder

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	杨长沛, 廖列法. 基于门控空洞卷积特征融合的中文命名实体识别[J]. 计算机工程, 2023, 49(8): 85-95.
[2]	张家熔, 苑津莎, 许珈宁, 罗志宏. 基于多元信息嵌入与协同神经网络的力学实体识别算法[J]. 计算机工程, 2023, 49(7): 125-134.
[3]	曹坪, 杨怀志, 薄一军, 尤嘉, 张淳杰, 李丹勇. 面向低质量裂缝图像的多知识蒸馏分类[J]. 计算机工程, 2023, 49(7): 204-213.
[4]	白明昌. 基于折叠路径聚合的属性网络节点嵌入方法[J]. 计算机工程, 2023, 49(7): 76-84.
[5]	刘豪, 吴红兰, 房宇轩. 结合全局上下文信息的高效人体姿态估计[J]. 计算机工程, 2023, 49(7): 102-109.
[6]	王款, 宣士斌, 何雪东, 李紫薇, 李嘉祥. 基于交叉注意力Transformer的人体姿态估计方法[J]. 计算机工程, 2023, 49(7): 223-231.
[7]	陈明, 刘蓉, 张晔. 基于多重注意力机制的中文医疗实体识别[J]. 计算机工程, 2023, 49(6): 314-320.
[8]	代祖华, 刘园园, 狄世龙. 语义增强的图神经网络方面级文本情感分析[J]. 计算机工程, 2023, 49(6): 71-80.
[9]	沈学利, 田桂源, 姜彦吉, 马琳琳. 基于双阶段Conv-Transformer的时频域语音增强算法[J]. 计算机工程, 2023, 49(6): 123-130.
[10]	丁子轩, 俞雷, 张娟, 李想, 王新宇. 基于深度残差自适应注意力网络的图像超分辨率重建[J]. 计算机工程, 2023, 49(5): 231-238.
[11]	毛亮, 赵林均, 余敦辉, 孙斌. 基于知识蒸馏的企业命名实体识别模型[J]. 计算机工程, 2023, 49(5): 90-96.
[12]	朱红, 牛浩然, 朱彤. 基于字词融合与对抗训练的行业人物实体识别[J]. 计算机工程, 2023, 49(5): 56-62.
[13]	陈治旭, 靳雁霞, 芦烨, 杨晶, 刘亚变, 史志儒. 基于子图卷积神经网络的多精度服装建模方法[J]. 计算机工程, 2023, 49(4): 174-181.
[14]	徐康, 李霏, 姬东鸿. 结合依存图卷积与文本片段搜索的方面情感三元组抽取[J]. 计算机工程, 2023, 49(4): 61-67.
[15]	衡红军, 苗菁. 语义与句法信息加强的二元标记实体关系联合抽取[J]. 计算机工程, 2023, 49(4): 77-84.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于CNN‐Head Transformer编码器的中文命名实体识别

Chinese Named Entity Recognition Based on CNN-Head Transformer Encoder

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献

相关文章 15

编辑推荐

Metrics

本文评价