Multidimensional Feature Named Entity Recognition Method in Education Domain

doi:10.19678/j.issn.1000-3428.0068669

Abstract

Abstract:

The development and progress of information technology have resulted in extensive investigations into ″Internet + Education″ in the field of education, and all aspects of education and teaching are being developed in the direction of intelligence. The study of Named Entity Recognition (NER) in secondary school mathematics can provide a foundation for the subsequent construction of secondary school mathematics knowledge mapping and automatic question-and-answer tasks to fulfill the demands of secondary school students for personalized knowledge acquisition and facilitate the construction of a new intelligent education system. Currently, owing to the semantic complexity of secondary school mathematics knowledge, its NER and research data are insufficient, and the current mainstream model for feature extraction disregards some local features. To solve the challenges of entity recognition in this field, a multidimensional feature NER method incorporating multihead attention is proposed using a self-constructed corpus of secondary school mathematics knowledge. First, the method adopts Bidirectional Encoder Representations from Transformers (BERT) for pre-training text representations to obtain word vectors. Subsequently, this method introduces adversarial training to perturb each embedding vector and then transmits the obtained adversarial samples and embedding vectors to the multidimensional feature extraction layer for feature extraction. Next, it splices the output features, dynamically fuses them via the multihead attention mechanism, and finally outputs them after correction by a Conditional Random Field (CRF). Experimental results show that the accuracy, recall, and F1 value of this method for recognizing the self-constructed Educ dataset are 96.68%, 97.71%, and 97.19%, respectively, thus demonstrating its effectiveness in recognizing mathematical knowledge entities in secondary schools.

Key words: Named Entity Recognition(NER), educational domain, adversarial training, multidimensional feature extraction, multi-head attention mechanism

摘要：

信息技术的发展与进步促使“互联网+教育”成为目前教育领域的研究热点, 教育教学的各个环节都在向智能化的方向发展。中学数学的命名实体识别(NER)任务的研究, 可为后续构建中学数学学科知识图谱及自动问答等任务奠定基础, 进而满足中学生个性化知识获取的需求, 助力新型智能化教育体系的构建。目前中学数学知识语义复杂, 其NER和研究数据较少, 且在当前主流模型特征提取任务中容易忽略掉部分局部特征。为解决该领域的实体识别困难问题, 以自建的中学数学知识语料库为研究对象, 提出一种融合多头注意力的多维度特征NER方法。该方法首先采用BERT进行文本表征预训练得到词向量, 接着引入对抗训练对每个嵌入向量进行扰动, 将得到的对抗样本和嵌入向量传送到多维度特征提取层进行特征提取, 再将输出的特征进行拼接, 通过多头注意力机制进行动态融合, 最终经过条件随机场(CRF)修正后输出。实验结果表明, 该方法在自建Educ数据集上的识别准确率、召回率以及F1值分别达到96.68%、97.71%和97.19%, 证明了该方法在中学数学知识实体识别上的有效性。

关键词: 命名实体识别, 教育领域, 对抗训练, 多维度特征提取, 多头注意力机制

REN Yi, SU Bo, YUAN Shuai. Multidimensional Feature Named Entity Recognition Method in Education Domain[J]. Computer Engineering, 2024, 50(10): 110-118.

任义, 苏博, 袁帅. 教育领域下多维度特征命名实体识别方法[J]. 计算机工程, 2024, 50(10): 110-118.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0068669

https://www.ecice06.com/EN/Y2024/V50/I10/110

Figures/Tables 17

Fig.1 Example of sequence labeling

Fig.2 Multidimensional feature named entity recognition model incorporating multi-head attention

Fig.3 Network structure of BERT

Fig.4 Input profiling of BERT

Fig.5 Multidimensional feature extraction layer structure

Fig.6 Schematic drawing of expansion convolution

Fig.7 Structure of multi-head attention mechanism network

References 23

1	涂冬波, 蔡艳, 丁树良. 认知诊断理论、方法与应用. 北京: 北京师范大学出版社, 2012.
	TU D B , CAI Y , DING S L . Cognitive diagnosis: theory, methods and applications. Beijing: Beijing Normal University Press, 2012.
2	LI J , SUN A X , HAN J L , et al. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering, 2022, 34 (1): 50- 70. doi: 10.1109/TKDE.2020.2981314
3	BI M W , ZHANG Q C , ZUO M , et al. Bi-directional long short-term memory model with semantic positional attention for the question answering system. ACM Transactions on Asian and Low-Resource Language Information Processing, 2021, 20 (5): 1- 13.
4	赵京胜, 宋梦雪, 高祥, 等. 自然语言处理中的文本表示研究. 软件学报, 2022, 33 (1): 102- 128.
	ZHAO J S , SONG M X , GAO X , et al. Research on text representation in natural language processing. Journal of Software, 2022, 33 (1): 102- 128.
5	张毅, 王爽胜, 何彬, 等. 基于BERT的初等数学文本命名实体识别方法. 计算机应用, 2022, 42 (2): 433- 439.
	ZHANG Y , WANG S S , HE B , et al. Named entity recognition method of elementary mathematical text based on BERT. Journal of Computer Applications, 2022, 42 (2): 433- 439.
6	HOCHREITER S , SCHMIDHUBER J . Long short-term memory. Neural Computation, 1997, 9 (8): 1735- 1780. doi: 10.1162/neco.1997.9.8.1735
7	刘世泽, 秦艳君, 王晨星, 等. 基于多尺度特征提取的交通模式识别算法. 计算机应用, 2021, 41 (6): 1573- 1580.
	LIU S Z , QIN Y J , WANG C X , et al. Transportation mode recognition algorithm based on multi-scale feature extraction. Journal of Computer Applications, 2021, 41 (6): 1573- 1580.
8	DEVLIN J, CHANG M W, LEE K, et al. Bert: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2023-09-10]. https://arxiv.org/abs/1810.04805.
9	刘斐, 文中, 吴艺. 基于BERT-BILSTM-CRF模型的电力行业事故文本智能分析. 中国安全生产科学技术, 2023, 19 (1): 209- 215.
	LIU F , WEN Z , WU Y . Intelligent analysis on text of power industry accident based on BERT-BILSTM-CRF model. Journal of Safety Science and Technology, 2023, 19 (1): 209- 215.
10	肖勇, 郑楷洪, 王鑫, 等. 基于联合神经网络学习的中文电力计量命名实体识别. 浙江大学学报(理学版), 2021, 48 (3): 321- 330.
	XIAO Y , ZHENG K H , WANG X , et al. Chinese named entity recognition in electric power metering domain based on neural joint learning. Journal of Zhejiang University (Science Edition), 2021, 48 (3): 321- 330.
11	曾青霞, 熊旺平, 杜建强, 等. 结合自注意力的BiLSTM-CRF的电子病历命名实体识别. 计算机应用与软件, 2021, 38 (3): 159-162, 234.
	ZENG Q X , XIONG W P , DU J Q , et al. Electronic medical record named entity recognition combined with self-attention bilstm-crf. Computer Applications and Software, 2021, 38 (3): 159-162, 234.
12	廖涛, 勾艳杰, 张顺香. 融合注意力机制的BERT-BiLSTM-CRF中文命名实体识别. 阜阳师范大学学报(自然科学版), 2021, 38 (3): 86- 91.
	LIAO T , GOU Y J , ZHANG S X . BERT-BiLSTM-CRF Chinese named entity recognition combined with attention mechanism. Journal of Fuyang Normal University (Natural Science), 2021, 38 (3): 86- 91.
13	AN Y , XIA X Y , CHEN X L , et al. Chinese clinical named entity recognition via multi-head self-attention based BiLSTM-CRF. Artificial Intelligence in Medicine, 2022, 127, 102282. doi: 10.1016/j.artmed.2022.102282
14	ZHANG Q H , WU M , LÜ P T , et al. Research on named entity recognition of Chinese electronic medical records based on multi-head attention mechanism and character-word information fusion. Journal of Intelligent & Fuzzy Systems, 2022, 42 (4): 4105- 4116.
15	苏博. 教育领域中学数学NER数据集[EB/OL]. [2023-09-10]. https://blog.csdn.net/sooxiaober/article/details/134898790.
	SU B. NER dataset for middle school mathematics in the field of education [EB/OL]. [2023-12-10]. https://blog.csdn.net/sooxiaober/article/details/134898790. (in Chinese)
16	杨长沛, 廖列法. 基于门控空洞卷积特征融合的中文命名实体识别. 计算机工程, 2023, 49 (8): 85- 95.
	YANG C P , LIAO L F . Chinese named entity recognition based on dilated gated convolution feature fusion. Computer Engineering, 2023, 49 (8): 85- 95.
17	赵建强, 朱万彤, 陈诚. 基于多重卷积神经网络模型的命名实体识别. 计算机技术与发展, 2023, 33 (1): 187- 192.
	ZHAO J Q , ZHU W T , CHEN C . Named entity recognition based on duplex convolution neural network model. Computer Technology and Development, 2023, 33 (1): 187- 192.
18	YANG Z, DAI Z, YANG Y, et al. XLNet: generalized autoregressive pretraining for language understanding[C]//Proceedings of Neural Information Processing Systems. Cambridge, USA: MIT Press, 2019: 5753-5763.
19	LAN Z H, CHEN M D, GOODMAN S, et al. ALBERT: a lite BERT for self-supervised learning of language representations[C]//Proceedings of the 8th International Conference on Learning Representations. Washington D. C., USA: IEEE Press, 2020: 1-17.
20	LIU Y, OTT M, GOYAL N, et al. RoBERTa: a robustly optimized BERT pretraining approach[EB/OL]. [2023-09-10]. https://arxiv.org/abs/1907.11692.
21	方美丽, 郑莹莹, 陶坤旺, 等. 基于MacBERT和对抗训练的城市内涝信息识别方法. 集成技术, 2023, 12 (1): 56- 67.
	FANG M L , ZHENG Y Y , TAO K W , et al. Urban waterlogging information recognition method based on MacBERT and adversarial training. Journal of Integration Technology, 2023, 12 (1): 56- 67.
22	LEVOW G. The third international Chinese language processing bakeoff: word segmentation and named entity recognition[C]//Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing. Sydney, Australia: Association for Computational Linguistics, 2006: 108-117.
23	GUI T, MA R T, ZHANG Q, et al. CNN-based Chinese NER with lexicon rethinking[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence. Washington D. C., USA: IEEE Press, 2019: 4982-4988.

[1]	WANG Yanguo, LÜ Pengyuan, LAN Jinjiang, LIU Mingzhe, QIN Guanjun, ZHANG Shuohua, ZHOU Yu. Wind Turbine Fault Classification Method Based on Adversarial Training and Transformer [J]. Computer Engineering, 2024, 50(9): 377-384.
[2]	DANG Xiaochao, LIU Jian, DONG Xiaohui, ZHU Zhongyan, LI Fenfang. Named Entity Recognition of Mechanical Equipment Failure for Imbalanced Data [J]. Computer Engineering, 2024, 50(9): 104-112.
[3]	Huayu LI, Zhikang ZHANG, Yang YAN, Yang YUE. Enhanced Domain Multi-modal Entity Recognition Based on Knowledge Graph [J]. Computer Engineering, 2024, 50(8): 31-39.
[4]	Huaqing ZHANG, Zhangtao XIA, Xiaoqing LU, Jijun TONG. Named Entity Recognition of Vascular Surgery Based on Glyph Features [J]. Computer Engineering, 2024, 50(8): 13-21.
[5]	WEI Zhuoyi, LUO Mai, LI Wenbing, ZENG Yuansong, YU Weijiang, YANG Yuedong. Intelligent Single-Cell Classification Based on Multisource Domain Adaptation [J]. Computer Engineering, 2024, 50(6): 48-55.
[6]	WEI Hao, DIAO Hongyue, KONG Liangchen, DENG Yaochen. Research on Fine-grained Named-Entity-Recognition Method for Public-Opinion Texts in Northeast Asia [J]. Computer Engineering, 2024, 50(5): 354-362.
[7]	Hao WEI, Hongyue DIAO, Liangchen KONG, Yaochen DENG. Research on Fine-grained Named-Entity-Recognition Method for Public-Opinion Texts in Northeast Asia [J]. Computer Engineering, 2024, 50(5): 354-362.
[8]	DAI Wei, WANG Fengyu, JI Changpeng. Aspect Level Sentiment Analysis Based on Sentiment-Enhanced and Dual Graph Convolutional Network [J]. Computer Engineering, 2024, 50(5): 120-127.
[9]	ZHOU Zhaochen, FANG Qingmao, WU Xiaohong, HU Ping, HE Xiaohai. Machine Reading Comprehension Model Based on MacBERT and Adversarial Training [J]. Computer Engineering, 2024, 50(5): 41-50.
[10]	Zhidong SHEN, Hengxian YUE. Textual Adversarial Training Method Based on Distributed Perturbation [J]. Computer Engineering, 2023, 49(9): 16-22.
[11]	Changpei YANG, Liefa LIAO. Chinese Named Entity Recognition Based on Dilated Gated Convolution Feature Fusion [J]. Computer Engineering, 2023, 49(8): 85-95.
[12]	Jiarong ZHANG, Jinsha YUAN, Jianing XU, Zhihong LUO. Mechanics Entities Recognition Algorithm Based on Multi-Meta Information Embedding and Collaborative Neural Network [J]. Computer Engineering, 2023, 49(7): 125-134.
[13]	CHEN Ming, LIU Rong, ZHANG Ye. Chinese Medical Entity Recognition Based on Multiple Attention Mechanism [J]. Computer Engineering, 2023, 49(6): 314-320.
[14]	ZHU Hong, NIU Haoran, ZHU Tong. Entity Recognition of Industry Figures Based on Character and Word Fusion and Adversarial Training [J]. Computer Engineering, 2023, 49(5): 56-62.
[15]	LIAO Liefa, XIE Shusong. Chinese Named Entity Recognition Based on Attention Mechanism Feature Fusion [J]. Computer Engineering, 2023, 49(4): 256-262.

Please choose a citation manager

Content to export