Biomedical Named Entity Recognition Method Based on Word Meaning Enhancement

doi:10.19678/j.issn.1000-3428.0065880

Abstract

Abstract:

Biomedical Named Entity Recognition(BioNER), as a core task of biomedical text mining, provides strong support for downstream tasks. There are more unregistered words in biomedical data than in the general domain. Existing BioNER methods usually use the method of splitting unregistered words into morphemes to alleviate the problem of insufficient information of unregistered words; however, the internal information of words is also split, resulting in label inconsistency. Thus, cross-entity label problems are prone to occur in label prediction for morphemes. In addition, the segmentation of words into morphemes leads to longer sentence lengths, which aggravates the problem of gradient disappearance during training. To address the above problems, a BioNER method that uses the Bidirectional Long Short-Term Memory(BiLSTM)-Biaffine structure is proposed for word meaning enhancement. First, morpheme representation information is obtained through the BioBERT pre-training model. Subsequently, BiLSTM-Biaffine is used to enhance the word sense, with BiLSTM at the word level to obtain forward and backward sequence information of the morpheme and the Biaffine attention mechanism to enhance the associated information and reintegrate it into the words representation. Finally, the label sequence of the input sentence is obtained through the BiLSTM-CRF model. The experimental results show that on the BC2GM, NCBI-Disease, BC5CDR-chem, and JNLPBA datasets, the F1 scores of the method reached 84.94%, 89.07%, 92.14%, and 74.57%, respectively. Compared with mainstream sequence annotation models such as the MTM-CW and MT-BioNER, the proposed method provided an average improvement of 2.99, 1.84, 3.09, and 1.03 percentage points, respectively, verifying its effectiveness in BioNER tasks.

Key words: Biomedical Named Entity Recognition(BioNER), morpheme, word meaning enhancement, Bidirectional Long Short-Term Memory (BiLSTM) network, attention mechanism

摘要：

生物医学命名实体识别(BioNER)是生物医学文本挖掘的核心任务之一，能够为下游任务提供有力支撑。与通用领域相比，生物医学数据中存在更多的未登录词，现有BioNER方法通常将未登录词拆分为语素进行表示学习，这种方法缓解了未登录词表示信息不足的问题，但是破坏了单词的内部信息，对语素进行标签预测时容易出现标签不一致和跨实体标签问题。此外，将单词分割为语素导致句子长度变长，加重了训练中存在的梯度消失问题。提出一种通过BiLSTM-Biaffine结构进行词义增强的BioNER方法。通过BioBERT预训练模型获取语素表示信息，使用BiLSTM-Biaffine进行词义增强，在单词层面利用BiLSTM分别获取语素的前向和后向序列信息，采用Biaffine注意力机制增强其关联信息并重新融合为单词表示，最后通过BiLSTM-CRF模型获取输入句子的标签序列。实验结果表明，在数据集BC2GM、NCBI-Disease、BC5CDR-chem和JNLPBA上，该方法的F1值分别达到84.94%、89.07%、92.14%和74.57%, 与主流序列标注模型MTM-CW、MT-BioNER等相比平均分别提高了2.99、1.84、3.09和1.03个百分点，验证了所提方法在BioNER任务中的有效性。

关键词: 生物医学命名实体识别, 语素, 词义增强, 双向长短期记忆网络, 注意力机制

Mengxuan CHEN, Yanping CHEN, Ying HU, Ruizhang HUANG, Yongbin QIN. Biomedical Named Entity Recognition Method Based on Word Meaning Enhancement[J]. Computer Engineering, 2023, 49(10): 305-312.

陈梦萱, 陈艳平, 扈应, 黄瑞章, 秦永彬. 基于词义增强的生物医学命名实体识别方法[J]. 计算机工程, 2023, 49(10): 305-312.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0065880

http://www.ecice06.com/EN/Y2023/V49/I10/305

Figures/Tables 8

References 35

1	JIN Q, DHINGRA B, COHEN W, et al. Probing biomedical embeddings from language models[EB/OL]. [2022-08-05]. https://arxiv.org/pdf/1904.02181.pdf.
2	YUAN Z, LIU Y, TAN C, et al. Improving biomedical pretrained language models with knowledge[EB/OL]. [2022-08-05]. https://arxiv.org/abs/2104.10344.
3	ALSENTZER E, MURPHY J, BOAG W, et al. Publicly available clinical BERT embeddings[EB/OL]. [2022-08-05]. https://arxiv.org/abs/1904.03323.
4	PENG Y F, YAN S K, LU Z Y. Transfer learning in biomedical natural language processing: an evaluation of BERT and ELMo on ten benchmarking datasets[EB/OL]. [2022-08-05]. https://arxiv.org/abs/1906.05474.
5	BELTAGY I, LO K, COHAN A. SciBERT: a pretrained language model for scientific text[EB/OL]. [2022-08-05]. https://arxiv.org/abs/1903.10676.
6	LEE J, YOON W, KIM S, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 2020, 36 (4): 1234- 1240. doi: 10.1093/bioinformatics/btz682
7	LAFFERTY J D, MCCALLUM A, PEREIRA F C N. Conditional random fields: probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the 18th International Conference on Machine Learning. New York, USA: ACM Press, 2001: 282-289.
8	YOON K. Convolutional neural networks for sentence classification[EB/OL]. [2022-08-05]. https://arxiv.org/abs/1408.5882.
9	HOCHREITER S, SCHMIDHUBER J. Long short-term memory. Neural Computation, 1997, 9 (8): 1735- 1780. doi: 10.1162/neco.1997.9.8.1735
10	BHASURAN B, MURUGESAN G, ABDULKADHAR S, et al. Stacked ensemble combined with fuzzy matching for biomedical named entity recognition of diseases. Journal of Biomedical Informatics, 2016, 64, 1- 9. doi: 10.1016/j.jbi.2016.09.009
11	YAO L, LIU H, LIU Y, et al. Biomedical named entity recognition based on deep neutral network. International Journal of Hybrid Information Technology, 2015, 8 (8): 279- 288. doi: 10.14257/ijhit.2015.8.8.29
12	LI L S, JIN L K, JIANG Z C, et al. Biomedical named entity recognition based on extended recurrent neural networks[C]//Proceedings of IEEE International Conference on Bioinformatics and Biomedicine. Washington D. C., USA: IEEE Press, 2015: 649-652.
13	LIMSOPATHAM N, COLLIER N. Learning orthographic features in bi-directional LSTM for biomedical named entity recognition[C]//Proceedings of the 5th Workshop on Building and Evaluating Resources for Biomedical Text Mining. Washington D. C., USA: IEEE Press, 2016: 10-19.
14	CRICHTON G, PYYSALO S, CHIU B, et al. A neural network multi-task learning approach to biomedical named entity recognition. BMC Bioinformatics, 2017, 18 (1): 1- 14. doi: 10.1186/s12859-016-1414-x
15	GREENBERG N, BANSAL T, VERGA P, et al. Marginal likelihood training of BiLSTM-CRF for biomedical named entity recognition from disjoint label sets[EB/OL]. [2022-08-05]. https://aclanthology.org/D18-1306.pdf.
16	CHO H, LEE H. Biomedical named entity recognition using deep neural networks with contextual information. BMC Bioinformatics, 2019, 20 (1): 1- 11. doi: 10.1186/s12859-018-2565-8
17	YOON W, SO C H, LEE J, et al. CollaboNet: collaboration of deep neural networks for biomedical named entity recognition. BMC Bioinformatics, 2019, 20 (10): 55- 65.
18	WANG X, ZHANG Y, REN X, et al. Cross-type biomedical named entity recognition with deep multi-task learning. Bioinformatics, 2019, 35 (10): 1745- 1752. doi: 10.1093/bioinformatics/bty869
19	WANG X, LÜ J G, DONG L, et al. Multitask learning for biomedical named entity recognition with cross-sharing structure. BMC Bioinformatics, 2019, 20 (1): 1- 13. doi: 10.1186/s12859-018-2565-8
20	HONG S K, LEE J G. DTranNER: biomedical named entity recognition with deep learning-based label-label transition model. BMC Bioinformatics, 2020, 21 (1): 1- 11. doi: 10.1186/s12859-019-3325-0
21	PHAM T H, LE-HONG P. End-to-end recurrent neural network models for Vietnamese named entity recognition: word-level vs. character-level[EB/OL]. [2022-08-05]. https://arxiv.org/abs/1705.04044.
22	GRIDACH M. Character-level neural network for biomedical named entity recognition. Journal of Biomedical Informatics, 2017, 70, 85- 91. doi: 10.1016/j.jbi.2017.05.002
23	LÜ C, CHEN B, REN Y F, et al. Long short-term memory RNN for biomedical named entity recognition. BMC Bioinformatics, 2017, 18 (1): 1- 11. doi: 10.1186/s12859-016-1414-x
24	李丽双, 郭元凯. 基于CNN-BLSTM-CRF模型的生物医学命名实体识别. 中文信息学报, 2018, 32 (1): 116- 122. doi: 10.3969/j.issn.1003-0077.2018.01.015
	LI L S, GUO Y K. Biomedical named entity recognition with CNN-BLSTM-CRF. Journal of Chinese Information Processing, 2018, 32 (1): 116- 122. doi: 10.3969/j.issn.1003-0077.2018.01.015
25	DANG T H, LE H Q, NGUYEN T M, et al. D3NER: biomedical named entity recognition using CRF-BiLSTM improved with fine-tuned embeddings of various linguistic information. Bioinformatics, 2018, 34 (20): 3539- 3546. doi: 10.1093/bioinformatics/bty356
26	CHO M, HA J, PARK C, et al. Combinatorial feature embedding based on CNN and LSTM for biomedical named entity recognition. Journal of Biomedical Informatics, 2020, 103, 103381. doi: 10.1016/j.jbi.2020.103381
27	TIAN Y H, SHEN W, SONG Y, et al. Improving biomedical named entity recognition with syntactic information. BMC Bioinformatics, 2020, 21 (1): 1- 17. doi: 10.1186/s12859-019-3325-0
28	许力, 李建华. 基于BERT和BiLSTM-CRF的生物医学命名实体识别. 计算机工程与科学, 2021, 43 (10): 1873- 1879. doi: 10.3969/j.issn.1007-130X.2021.10.022
	XU L, LI J H. Biomedical named entity recognition based on BERT and BiLSTM-CRF. Computer Engineering and Science, 2021, 43 (10): 1873- 1879. doi: 10.3969/j.issn.1007-130X.2021.10.022
29	于祥钦, 王香, 李智强, 等. 基于字符级特征自适应的生物医学命名实体识别. 小型微型计算机系统, 2023, 44 (9): 1876- 1883. doi: 10.20009/j.cnki.21-1106/TP.2021-0993
	YU X Q, WANG X, LI Z Q, et al. Biomedical named entity recognition based on character level feature adaptation. Journal of Chinese Computer Systems, 2023, 44 (9): 1876- 1883. doi: 10.20009/j.cnki.21-1106/TP.2021-0993
30	MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space[EB/OL]. [2022-08-05]. https://arxiv.org/abs/1301.3781.
31	PENNINGTON J, SOCHER R, MANNING C. GloVe: global vectors for word representation[C]//Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing. [S. l. ]: Association for Compu-tational Linguistics, 2014: 1532-1543.
32	PETERS M, NEUMANN M, IYYER M, et al. Deep contextualized word representations[EB/OL]. [2022-08-05].https://arxiv.org/abs/1802.05365.
33	DEVLIN J, CHANG M, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2022-08-05].https://arxiv.org/abs/1810.04805.
34	KHAN M R, ZIYADI M, ABDELHADY M. MT-BioNER: multi-task learning for biomedical named entity recognition using deep bidirectional transformers[EB/OL]. [2022-08-05].https://arxiv.org/abs/2001.08904.
35	SUN C, YANG Z, WANG L, et al. Biomedical named entity recognition using BERT in the machine reading comprehension framework. Journal of Biomedical Informatics, 2021, 118, 103799. doi: 10.1016/j.jbi.2021.103799

[1]	Xiaodong SU, Shizhou LI, Jiayuan ZHAO, Hongyu LIANG, Yurong ZHANG, Hongyan XU. Image Semantic Segmentation Based on Multi-level Superposition and Attention Mechanism [J]. Computer Engineering, 2023, 49(9): 265-271, 278.
[2]	Lu HAN, Weigang HUO, Yonghui ZHANG, Tao LIU. Multivariate Time Series Forecasting Based on Multi-Scale Feature Fusion and Dual-Attention Mechanism [J]. Computer Engineering, 2023, 49(9): 99-108.
[3]	Yujiang LONG, Wei WEI, Yu SHU, Zhenggang ZHANG, Daolei WANG, Feng LI. Detection Method for Damaged Rotating Insulator Based on Adaptive Key Points [J]. Computer Engineering, 2023, 49(9): 272-278.
[4]	Jing YANG, Minghua LU, Jieqiong MA, Jinping WU, Xingxuan LIU. Underwater Defense Posture Prediction Method Based on Alternating Recurrent Neural Network [J]. Computer Engineering, 2023, 49(9): 69-78.
[5]	Haoxin LIU, Chao DONG, Zhinan GOU, Kai GAO. Few-Shot Relation Extraction Method Fusing with Hybrid Representation [J]. Computer Engineering, 2023, 49(8): 63-68.
[6]	Changpei YANG, Liefa LIAO. Chinese Named Entity Recognition Based on Dilated Gated Convolution Feature Fusion [J]. Computer Engineering, 2023, 49(8): 85-95.
[7]	Junhao LIU, Meilin WANG, Xing XIE, Yexing SONG, Lihua XU. Leather Defect Detection Algorithm Based on Improved YOLOv5 [J]. Computer Engineering, 2023, 49(8): 240-249.
[8]	Na MA, Tingxin WEN, Xu JIA, Xiaohui LI. Adaptive Vehicle Face Re-identification Model Under Complex Illumination Conditions [J]. Computer Engineering, 2023, 49(8): 275-282, 290.
[9]	Lumeng CHEN, Yanyan CAO, Min HUANG, Xingang XIE. Flame Detection Method Based on Improved YOLOv5 [J]. Computer Engineering, 2023, 49(8): 291-301, 309.
[10]	Qianglong LI, Xinwen ZHOU, Meng'en WEI, Yangzhou GAN. Infrared Target Detection Algorithm Based on Strip Pooling and Attention Mechanism in Street Scene [J]. Computer Engineering, 2023, 49(8): 310-320.
[11]	Shupeng WANG, Yindi HE. Uneven Illumination Image Enhancement Algorithm Fusing Feature Attention Mechanism [J]. Computer Engineering, 2023, 49(8): 232-239.
[12]	Fangyu FENG, Xiaoshu LUO, Zhiming MENG, Guangyu WANG. Facial Expression Recognition Based on Anti-Aliasing Residual Attention Network [J]. Computer Engineering, 2023, 49(8): 190-198.
[13]	Mingchang BAI. Node Embedding Method Based on Folded Path Aggregation on Attributed Network [J]. Computer Engineering, 2023, 49(7): 76-84.
[14]	Shan WU, Feng ZHOU. Small Target Detection Based on Improved SSD Algorithm [J]. Computer Engineering, 2023, 49(7): 179-188.
[15]	Yongsheng QI, Xiaoxu DU, Junfeng ZHU, Shengli GAO, Liqiang LIU. Efficient Livestock Detection in Grazing Areas Based on Enhanced Lightweight Deep Network [J]. Computer Engineering, 2023, 49(7): 278-287.

Please choose a citation manager

Content to export