基于多元信息嵌入与协同神经网络的力学实体识别算法

doi:10.19678/j.issn.1000-3428.0065117

摘要/Abstract

摘要：

自动抽取力学问题中的关键实体是力学问题自动解答的重要手段。然而，与开放域相比，力学问题具有大量的专业词汇和较长的实体，其识别难度高，准确率低。针对该问题，基于图注意力网络（GAT）和Transformer编码器提出一种用于力学问题关键实体抽取的实体识别算法。针对汉语的特点设计一个包括字符信息、词汇信息和部首信息的多元信息嵌入用于增强中文句子表示。提出结构图和语境图两个图模型对中文句子进行建模，并设计一种协同架构，该架构使用两个独立的GAT整合多元信息并学习句子的上下文信息。为平衡词汇信息与部首信息对中文字符的影响，提出一种协同Transformer架构，该架构由字符-词汇Transformer与字符-部首Transformer构成，并增加词汇-部首注意力偏置，从而增强模型的识别能力。在自建的数据集上进行多组对比实验，结果表明，在力学问题实体识别任务中，相对于WCGCN算法，该算法在精度、召回率和F1值上分别提高1.92、0.99和1.44个百分点，能够有效提取力学问题中的关键信息。

关键词: 命名实体识别, 多元信息嵌入, 图注意力网络, Transformer架构, 注意力机制

Abstract:

Automatic extraction of key entities is important for solving mechanical problems automatically. However, compared to open domains, mechanical problems entail a larger specialized vocabulary and longer entities, making their recognition difficult and thus lowering the accuracy. To address this issue, an entity recognition algorithm based on a Graph Attention Network (GAT) and Transformer encoder is proposed for key entity extraction in mechanical problems. A Multi-Meta Information Embedding (MMIE) that includes character information, lexical information, and radical information is designed to enhance Chinese sentence representation based on the characteristics of Chinese. Additionally, two graph models, Structural Graph (S-Graph) and Contextual Graph (C-Graph), are proposed to model Chinese sentences. To learn the contextual information of sentences, a collaborative architecture is designed that integrates multiple types of information using two independent GATs. To balance the impact of lexical information and radical information on Chinese characters, a collaborative Transformer architecture is proposed, consisting of a character-lexical Transformer and a character-radical Transformer. A lexical-radical attention bias is added to enhance the recognition ability of the model. Several comparative experiments were conducted on a self-built dataset, and the results show that the proposed algorithm improves precision, recall, and F1 values by 1.92, 0.99, and 1.44 percentage points compared to Word-Character Graph Convolution Network(WCGCN) algorithm, respectively, for the entity recognition task in mechanical problems, and the algorithm effectively extracts key information in mechanical problems.

Key words: Named Entity Recognition(NER), Multi-Meta Information Embedding(MMIE), Graph Attention Network(GAT), Transformer architecture, attention mechanism

张家熔, 苑津莎, 许珈宁, 罗志宏. 基于多元信息嵌入与协同神经网络的力学实体识别算法[J]. 计算机工程, 2023, 49(7): 125-134.

Jiarong ZHANG, Jinsha YUAN, Jianing XU, Zhihong LUO. Mechanics Entities Recognition Algorithm Based on Multi-Meta Information Embedding and Collaborative Neural Network[J]. Computer Engineering, 2023, 49(7): 125-134.

https://www.ecice06.com/CN/Y2023/V49/I7/125

图/表 15

参考文献 34

1	WANG W Y, PAN S J. Integrating deep learning with logic fusion for information extraction. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(5): 9225- 9232. doi: 10.1609/aaai.v34i05.6460
2	ZHANG N Y, YE H B, DENG S M, et al. Contrastive information extraction with generative Transformer. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29, 3077- 3088. doi: 10.1109/TASLP.2021.3110126
3	LAI T Q, CHENG L L, WANG D P, et al. RMAN: relational multi-head attention neural network for joint extraction of entities and relations. Applied Intelligence, 2022, 52(3): 3132- 3142. doi: 10.1007/s10489-021-02600-2
4	车海燕, 冯铁, 张家晨, 等. 面向中文自然语言文档的自动知识抽取方法. 计算机研究与发展, 2013, 50(4): 834- 842. URL
	CHE H Y, FENG T, ZHANG J C, et al. Automatic knowledge extraction from Chinese natural language documents. Journal of Computer Research and Development, 2013, 50(4): 834- 842. URL
5	ALANI H, KIM S, MILLARD D E, et al. Automatic ontology-based knowledge extraction from Web documents. IEEE Intelligent Systems, 2003, 18(1): 14- 21. doi: 10.1109/MIS.2003.1179189
6	DO P, PHAN T H V. Developing a BERT based triple classification model using knowledge graph embedding for question answering system. Applied Intelligence, 2022, 52(1): 636- 651. doi: 10.1007/s10489-021-02460-w
7	CHEN X C, YANG Z Y, LIANG N Y, et al. Co-attention fusion based deep neural network for Chinese medical answer selection. Applied Intelligence, 2021, 51(10): 6633- 6646. doi: 10.1007/s10489-021-02212-w
8	ZHANG J, YUAN J, GUO H, et al. Integrating deep learning with first order logic for solving kinematic problems. Applied Intelligence, 2022, 52(3): 11808- 11826.
9	HE B, YU X G, JIAN P P, et al. A relation based algorithm for solving direct current circuit problems. Applied Intelligence, 2020, 50(7): 2293- 2309. doi: 10.1007/s10489-020-01667-7
10	DUAN H, YAN Z. A study on features of the CRFs-based Chinese named entity recognition[EB/OL]. [2022-05-05]. https://www.semanticscholar.org/paper/A-Study-on-Features-of-the-CRFs-based-Chinese-Named-Duan/a874006d45beb668603e382a7fcf29f6cfe6baec.
11	VELIČKOVIĆ P, CUCURULL G, CASANOVA A, et al. Graph attention networks[EB/OL]. [2022-05-05]. https://arxiv.org/abs/1710.10903.
12	ASHISH V, NOAM S, NIKI P, et al. Named entity recognition for Chinese social media with jointly trained embeddings[EB/OL]. [2022-05-05]. https://aclanthology.org/D15-1064.pdf.
13	PENG N Y, DREDZE M. Improving named entity recognition for Chinese social media with word segmentation representation learning[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. [S.l.]: Association for Computational Linguistics, 2016: 149-155.
14	PENG N Y, DREDZE M. Named entity recognition for Chinese social media with jointly trained embeddings[C]//Proceedings of 2015 Conference on Empirical Methods in Natural Language Processing. [S.l.]: Association for Computational Linguistics, 2015: 548-554.
15	XUE M G, YU B W, LIU T W, et al. Porous lattice-based Transformer encoder for Chinese NER[EB/OL]. [2022-05-05]. https://arxiv.org/abs/1911.02733.
16	ZHU Y Y, WANG G X, KARLSSON B F. CAN-NER: convolutional attention network for Chinese named entity recognition[EB/OL]. [2022-05-05]. https://arxiv.org/pdf/1904.02141.pdf.
17	KONG Z Q, TANG B P, DENG L, et al. Condition monitoring of wind turbines based on spatio-temporal fusion of SCADA data by convolutional neural networks and gated recurrent units. Renewable Energy, 2020, 146, 760- 768. doi: 10.1016/j.renene.2019.07.033
18	YU J, XUN J, HAO X, et al. Joint embeddings of Chinese words, characters, and fine-grained subcharacter components[EB/OL]. [2022-05-05]. https://aclanthology.org/D17-1027.pdf.
19	MENG Y, WU W, WANG F, et al. Glyce: glyph-vectors for Chinese character representations[EB/OL]. [2022-05-05]. https://arxiv.org/abs/1901.10125.
20	SONG C J, XIONG Y, HUANG W C, et al. Joint self-attention and multi-embeddings for Chinese named entity recognition[C]//Proceedings of the 6th International Conference on Big Data Computing and Communications. Washington D. C., USA: IEEE Press, 2020: 76-80.
21	CHEN A G, YIN C L. CRW-NER: exploiting multiple embeddings for Chinese named entity recognition[C]//Proceedings of the 4th International Conference on Artificial Intelligence and Big Data. Washington D. C., USA: IEEE Press, 2021: 520-524.
22	ZHANG Y, YANG J. Chinese NER using Lattice LSTM[EB/OL]. [2022-05-05]. https://arxiv.org/pdf/1805.02023.pdf.
23	GUI T, ZOU Y C, PENG M L, et al. A lexicon-based graph neural network for Chinese NER[EB/OL]. [2022-05-05]. https://aclanthology.org/D19-1096.pdf.
24	TANG Z, WAN B Y, YANG L. Word-character graph convolution network for Chinese named entity recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020, 28, 1520- 1532. doi: 10.1109/TASLP.2020.2994436
25	DING R X, XIE P J, ZHANG X Y, et al. A neural multi-digraph model for Chinese NER with gazetteers[EB/OL]. [2022-05-05]. https://aclanthology.org/P19-1141.pdf.
26	GUI T, MA R T, ZHAO L J, et al. CNN-based Chinese NER with lexicon rethinking[EB/OL]. [2022-05-05]. https://www.ijcai.org/Proceedings/2019/0692.pdf.
27	WU F Z, LIU J X, WU C H, et al. Neural Chinese named entity recognition via CNN-LSTM-CRF and joint training with word segmentation[EB/OL]. [2022-05-05]. https://arxiv.org/pdf/1905.01964.pdf.
28	MA R, PENG M, ZHANG Q, et al. Simplify the usage of lexicon in Chinese NER[EB/OL]. [2022-05-05]. https://arxiv.org/abs/1908.05969.
29	DONG C H, ZHANG J J, ZONG C Q, et al. Character-based LSTM-CRF with radical-level features for Chinese named entity recognition[EB/OL]. [2022-05-05]. https://link.springer.com/content/pdf/10.1007/978-3-319-50496-4_20.pdf?pdf=inline%20link.
30	张志昌, 周侗, 张瑞芳, 等. 融合双向GRU与注意力机制的医疗实体关系识别. 计算机工程, 2020, 46(6): 296- 302. URL
	ZHANG Z C, ZHOU T, ZHANG R F, et al. Medical entity relation recognition combining bidirectional GRU and attention. Computer Engineering, 2020, 46(6): 296- 302. URL
31	顾亦然, 霍建霖, 杨海根, 等. 基于BERT的电机领域中文命名实体识别方法. 计算机工程, 2021, 47(8): 78-83, 92 URL
	GU Y R, HUO J L, YANG H G, et al. BERT-based Chinese named entity recognition method in motor field. Computer Engineering, 2021, 47(8): 78-83, 92 URL
32	何阳宇, 晏雷, 易绵竹, 等. 融合CRF与规则的老挝语军事领域命名实体识别方法. 计算机工程, 2020, 46(8): 297- 304. URL
	HE Y Y, YAN L, YI M Z, et al. Named entitiy recognition method for Laotian in military field combining CRF and rules. Computer Engineering, 2020, 46(8): 297- 304. URL
33	LAFFERTY J D, MCCALLUM A, PEREIRA F C N. Conditional random fields: probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the 18th International Conference on Machine Learning. New York, USA: ACM Press, 2001: 282-289.
34	KINGMA D P, BA J. Adam: a method for stochastic optimization[EB/OL]. [2022-05-05]. https://arxiv.org/abs/1412.6980.

汉字	首尾部首	部件构造
运	辶云	辶云
动	云力	云力
度	广又	广甘又
直	十且	十且一

汉字	首尾部首	部件构造
运	辶云	辶云
动	云力	云力
度	广又	广甘又
直	十且	十且一

词语	字符关系
词语	词头关系	词内关系	词尾关系
重力	重	—	力
位移	位	—	移
摩擦力	摩	擦	力
加速度	加	速	度

词语	字符关系
词语	词头关系	词内关系	词尾关系
重力	重	—	力
位移	位	—	移
摩擦力	摩	擦	力
加速度	加	速	度

数据属性	训练集	验证集	测试集
字符数	712 589	92 702	89 854
实体数	42 023	5 448	5 467
问题数	5 457	682	683
平均字符	130.58	135.92	131.56

选择文件类型/文献管理软件名称

选择包含的内容