Chinese Medical Entity Recognition Based on Attention Enhancement and Feature Fusion

doi:10.19678/j.issn.1000-3428.0067645

Abstract

Abstract:

To address problems such as single embedding forms, difficult boundary recognition, and insufficient use of semantic information in Chinese medical named entity recognition models based on character representation, an effective method is to inject lexical features at the bottom of Bret. This approach reduces the impact of word segmentation errors while utilizing word granularity semantic information. However, some low correlation words and noise are introduced when vocabulary information is injected, leading to attention distraction in the Bret model based on the attention mechanism. In addition, it is difficult to fully mine deep semantic information of Chinese characters by relying solely on word granularity. Therefore, this study proposes a Chinese medical entity recognition model based on attention enhancement and feature fusion. The sparse processing of the attention score matrix of words causes the model to focus on words with a high correlation, which can effectively reduce the interference of noisy words in the context. Simultaneously, Convolutional Neural Networks (CNNs) are used to extract the features of Chinese pronunciation and strokes, which are fused with the output features of the Bret model through an iterative attention feature fusion module and subsequently concatenated to the BiLSTM model to further mine the deep semantic information contained in characters. During the experiment, a large number of relevant medical corpora is collected using a crawler and other methods. Further, a medical field word vector library is trained and verified on the CCKS2017 and CCKS2019 datasets. The experimental results show that the F1 values of the model reach 94.90% and 89.37%, respectively, which are higher than those with current mainstream entity recognition models. Therefore, the proposed model exhibits higher recognition performance.

Key words: entity recognition, Chinese word segmentation, sparse attention, feature fusion, medical word vector library

摘要：

针对基于字符表示的中文医学领域命名实体识别模型嵌入形式单一、边界识别困难、语义信息利用不充分等问题, 一种非常有效的方法是在Bret底层注入词汇特征, 在利用词粒度语义信息的同时降低分词错误带来的影响, 然而在注入词汇信息的同时也会引入一些低相关性的词汇和噪声, 导致基于注意力机制的Bret模型出现注意力分散的情况。此外仅依靠字、词粒度难以充分挖掘中文字符深层次的语义信息。对此, 提出基于注意力增强与特征融合的中文医学实体识别模型, 对字词注意力分数矩阵进行稀疏处理, 使模型的注意力集中在相关度高的词汇, 能够有效减少上下文中的噪声词汇干扰。同时, 对汉字发音和笔画通过卷积神经网络(CNN)提取特征, 经过迭代注意力特征融合模块进行融合, 然后与Bret模型的输出特征进行拼接输入给BiLSTM模型, 进一步挖掘字符所包含的深层次语义信息。通过爬虫等方式搜集大量相关医学语料, 训练医学领域词向量库, 并在CCKS2017和CCKS2019数据集上进行验证, 实验结果表明, 该模型F1值分别达到94.90%、89.37%, 效果优于当前主流的实体识别模型, 具有更好的识别效果。

关键词: 实体识别, 中文分词, 注意力稀疏, 特征融合, 医学词向量库

Jintao WANG, Ang QIN, Yuan ZHANG, Yifei CHEN, Tingfeng WANG, Chenglin XIE, Gang ZOU. Chinese Medical Entity Recognition Based on Attention Enhancement and Feature Fusion[J]. Computer Engineering, 2024, 50(7): 324-332.

王晋涛, 秦昂, 张元, 陈一飞, 王廷凤, 谢承霖, 邹刚. 基于注意力增强与特征融合的中文医学实体识别[J]. 计算机工程, 2024, 50(7): 324-332.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0067645

https://www.ecice06.com/EN/Y2024/V50/I7/324

Figures/Tables 13

Fig.1 Procedure of model construction

Fig.2 Filtering graph of Top-k attention scores

Fig.3 Module of multi-scale channels attention

Fig.4 LSMT unit structure

Fig.5 Radical distribution of CCKS2017

Fig.6 Radical distribution of CCKS2019

References 24

1	KE J, WANG W J, CHEN X J, et al. Medical entity recognition and knowledge map relationship analysis of Chinese EMRs based on improved BiLSTM-CRF. Computers and Electrical Engineering, 2023, 108, 108709. doi: 10.1016/j.compeleceng.2023.108709
2	杜晋华, 尹浩, 冯嵩. 中文电子病历命名实体识别的研究与进展. 电子学报, 2022, 50(12): 3030- 3053. URL
	DU J H, YIN H, FENG S. Research and development of named entity recognition in Chinese electronic medical record. Acta Electronica Sinica, 2022, 50(12): 3030- 3053. URL
3	CHEN C, KONG F. Enhancing entity boundary detection for better Chinese named entity recognition[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Washington D. C., USA: IEEE Press, 2021: 20-25.
4	COOK H V, JENDEN L J. A guide to dictionary-based text mining. Bioinformatics and Drug Discovery, 2019, 1939, 73- 89.
5	ZHANG Y, YANG J. Chinese NER using lattice LSTM[EB/OL]. [2023-04-10]. http://arxiv.org/abs/1805.02023v4.
6	LI Y, DU G D, XIANG Y, et al. Towards Chinese clinical named entity recognition by dynamic embedding using domain-specific knowledge. Journal of Biomedical Informatics, 2020, 106, 103435. doi: 10.1016/j.jbi.2020.103435
7	YIN M W, MOU C J, XIONG K N, et al. Chinese clinical named entity recognition with radical-level feature and self-attention mechanism. Journal of Biomedical Informatics, 2019, 98, 103289. doi: 10.1016/j.jbi.2019.103289
8	SHI J T, SUN M X, SUN Z Y, et al. Multi-level semantic fusion network for Chinese medical named entity recognition. Journal of Biomedical Informatics, 2022, 133, 104144. doi: 10.1016/j.jbi.2022.104144
9	ZHAO G X, LIN J Y, ZHANG Z Y, et al. Explicit sparse transformer: concentrated attention through explicit selection[EB/OL]. [2023-04-10]. http://arxiv.org/abs/1912.11637v1.
10	LIU W, FU X Y, ZHANG Y, et al. Lexicon enhanced Chinese sequence labeling using BERT adapter[EB/OL]. [2023-04-10]. http://arxiv.org/abs/2105.07148v3.
11	DAI Y, GIESEKE F, OEHMCKE S, et al. Attentional feature fusion[C]//Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision. Washington D. C., USA: IEEE Press, 2021: 3560-3569.
12	YANG X, HUANG W. A conditional random fields approach to biomedical named entity recognition. Journal of Electronics, 2007, 24(6): 838- 844.
13	LIU W, XU T, XU Q, et al. An encoding strategy based word-character LSTM for Chinese NER[C]//Proceedings of 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Washington D. C., USA: IEEE Press, 2019: 2379-2389.
14	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2023-04-10]. http://arxiv.org/abs/1810.04805v2.
15	SONG Y, SHI S, LI J, et al. Directional skip-gram: explicitly distinguishing left and right context for word embeddings[C]//Proceedings of 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Washington D. C., USA: IEEE Press, 2018: 175-180.
16	MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space[EB/OL]. [2023-04-10]. http://arxiv.org/abs/1301.3781v3.
17	LUO R X, XU J J, ZHANG Y, et al. PKUSEG: a toolkit for multi-domain Chinese word segmentation[EB/OL]. [2023-04-10]. http://arxiv.org/abs/1906.11455v3.
18	李军怀, 陈苗苗, 王怀军, 等. 基于ALBERT-BGRU-CRF的中文命名实体识别方法. 计算机工程, 2022, 48(6): 89-94, 106. URL
	LI J H, CHEN M M, WANG H J, et al. Chinese named entity recognition method based on ALBERT-BGRU-CRF. Computer Engineering, 2022, 48(6): 89-94, 106. URL
19	孙振, 李新福. 多特征融合的中文电子病历命名实体识别. 计算机工程与应用, 2023, 59(23): 136- 144. doi: 10.3778/j.issn.1002-8331.2207-0455
	SUN Z, LI X F. Named entity recognition of Chinese electronic medical records based on multi-feature fusion. Computer Engineering and Applications, 2023, 59(23): 136- 144. doi: 10.3778/j.issn.1002-8331.2207-0455
20	KONG J, ZHANG L X, JIANG M, et al. Incorporating multi-level CNN and attention mechanism for Chinese clinical named entity recognition. Journal of Biomedical Informatics, 2021, 116, 103737. doi: 10.1016/j.jbi.2021.103737
21	乔锐, 杨笑然, 黄文亢. 基于Bret与模型融合的医疗命名实体识别[C]//全国知识图谱与语义计算会议评估任务论文集. 杭州: [s. n.], 2019: 1-6.
	QIAO R, YANG X R, HUANG W K. Medical named entity recognition based on Bret and model fusion [C]//Proceedings of Evaluation Tasks at the China Conference on Knowledge Graph and Semantic Computing. Hangzhou: [s. n.], 2019: 1-6. (in Chinese)
22	AN Y, XIA X Y, CHEN X L, et al. Chinese clinical named entity recognition via multi-head self-attention based BiLSTM-CRF. Artificial Intelligence in Medicine, 2022, 127, 102282. doi: 10.1016/j.artmed.2022.102282
23	CAO J, ZHOU X, XIONG W, et al. Electronic medical record entity recognition via machine reading comprehension and biaffine. Discrete Dynamics in Nature and Society, 2021,(9): 1- 8.
24	WANG C Y, WANG H, ZHUANG H, et al. Chinese medical named entity recognition based on multi-granularity semantic dictionary and multimodal tree. Journal of Biomedical Informatics, 2020, 111, 103583. doi: 10.1016/j.jbi.2020.103583

[1]	DANG Xiaochao, LIU Jian, DONG Xiaohui, ZHU Zhongyan, LI Fenfang. Named Entity Recognition of Mechanical Equipment Failure for Imbalanced Data [J]. Computer Engineering, 2024, 50(9): 104-112.
[2]	LI Junyi, LI Xiangyang, LONG Chaoxun, LI Haiyan, LI Hongsong, YU Pengfei. Wild Mushroom Classification Based on Multi-level Region Selection and Cross-layer Feature Fusion [J]. Computer Engineering, 2024, 50(9): 179-188.
[3]	Wanqiu ZHAO, Junhu ZHANG, Haitao LI. Feature Fusion Network with Parallel Structure for Building Segmentation [J]. Computer Engineering, 2024, 50(8): 239-248.
[4]	Hong ZHAO, Xiao WANG. Study on Lesion Segmentation of Melanoma Images Based on Swin-Transformer [J]. Computer Engineering, 2024, 50(8): 249-258.
[5]	Li MIN, Bingjie DONG, Dong AN. Semantic Segmentation Algorithm Based on Multi-Attention Mechanism and Cross-Feature Fusion [J]. Computer Engineering, 2024, 50(8): 282-289.
[6]	Yuhang CHEN, Yong YANG, Xianmusiya·Maimaitiming, Palidan·Tuerxun, Xiaochao FAN, Ge REN, Yufeng DIAO. Automatic Essay Scoring Method Based on Topic Perception and Semantic Enhancement [J]. Computer Engineering, 2024, 50(8): 363-371.
[7]	Huaqing ZHANG, Zhangtao XIA, Xiaoqing LU, Jijun TONG. Named Entity Recognition of Vascular Surgery Based on Glyph Features [J]. Computer Engineering, 2024, 50(8): 13-21.
[8]	Huayu LI, Zhikang ZHANG, Yang YAN, Yang YUE. Enhanced Domain Multi-modal Entity Recognition Based on Knowledge Graph [J]. Computer Engineering, 2024, 50(8): 31-39.
[9]	Suolan LIU, Yan WANG, Hongyuan WANG, Shengsheng ZHU. Human Behavior Recognition Based on Multi-Stream Semantic Graph Convolutional Network [J]. Computer Engineering, 2024, 50(8): 64-74.
[10]	Juquan TAN, Ran WANG. Dynamic Time Warping Capture Algorithm for 3D Human Body Movements in Track and Field Video Recording Under Feature Fusion [J]. Computer Engineering, 2024, 50(7): 71-78.
[11]	Yiwen ZHANG, Manchun CAI, Yonghao CHEN, Yi ZHU, Lifeng YAO. Multi-Scale Deepfake Detection Method with Fusion of Spatial Features [J]. Computer Engineering, 2024, 50(7): 240-250.
[12]	LI Yakang, CHEN Gang. Automated Selection for Physical Models of Small-Angle Neutron Scattering [J]. Computer Engineering, 2024, 50(6): 56-64.
[13]	YANG Shuo, WANG Yiding. Facial Animation Algorithm Based on Improved Thin Plate Spline Motion Model [J]. Computer Engineering, 2024, 50(6): 255-265.
[14]	CAI Yixiang, QIN Pinle, ZENG Jianchao, JIN Zanxia, QIN Jia, ZHAI Shuangjiao. Research on Person Re-Identification Method for Large-Angle Viewpoint Differences [J]. Computer Engineering, 2024, 50(5): 330-341.
[15]	Hao WEI, Hongyue DIAO, Liangchen KONG, Yaochen DENG. Research on Fine-grained Named-Entity-Recognition Method for Public-Opinion Texts in Northeast Asia [J]. Computer Engineering, 2024, 50(5): 354-362.

Please choose a citation manager

Content to export