Distantly Supervised Relationship Extraction Combined with Residual BiLSTM and Sentence Bag Attention

doi:10.19678/j.issn.1000-3428.0062891

Abstract

Abstract: Entity relationship extraction entails identifying the semantic relationship between entities from unstructured and programmed massive texts and can provide data support for ontology construction, intelligent retrieval, and other tasks.However, the existing distantly supervised relationship extraction methods generally suffer from three problems:the need for numerous manual annotation corpora, noise contained in the extraction features, and the relationship between entities and sentences being ignored.This paper proposes a relationship extraction model based on a residual Bi-directional Long Short-Term Memory(BiLSTM) network and intra-sentence and inter-sentence bag attention mechanisms.Based on using word vectors and position vectors as model input, the model extracts long-distance text information in sentences and entity words through a residual BiLSTM network, and it uses the intra-sentence and inter-sentence bag attention mechanisms to process the extracted feature information.This enables the model to reduce the feature extraction noise between entities in the distantly supervision process and improve its recognition accuracy.The experimental results on the New York Times(NYT) dataset demonstrate that the model can make full use of the entity and relationship features, and the average precision reaches 86.2%.Compared with similar models using Convolutional Neural Network(CNN) and Piecewise Convolutional Neural Network(PCNN) as sentence encoders, the proposed model has better remote supervised relationship extraction performance.

Key words: natural language processing, relation extraction, residual connection, Bi-directional Long Short-Term Memory(BiLSTM) network, sentence bag attention mechanism

摘要： 实体关系抽取是从非结构化和程序化的海量文本中识别出实体之间的语义关系，为本体构建、智能检索等任务提供数据支持，然而现有远程监督关系抽取方法普遍存在需要大量人工标注语料库、提取特征含有噪声且忽略了实体与句子之间关联关系等问题。提出一种基于残差双向长短时记忆网络（BiLSTM）与句袋内和句袋间注意力机制的关系抽取模型，在将词向量和位置向量作为模型输入的基础上，通过残差BiLSTM网络提取语句与实体词中的长距离文本信息，利用句袋内和句袋间注意力机制对提取到的特征信息进行处理，使模型在远程监督过程中减少实体之间的特征提取噪声，并提高模型识别准确性。在NYT数据集上的实验结果表明，该模型能够充分利用实体与关系特征，平均精确率达到86.2%，相比于将卷积神经网络和分段卷积神经网络作为句子编码器的同类模型具有更好的远程监督关系抽取性能。

关键词: 自然语言处理, 关系抽取, 残差连接, 双向长短时记忆网络, 句袋注意力机制

CLC Number:

TP391.1

JIANG Xu, QIAN Xuezhong, SONG Wei. Distantly Supervised Relationship Extraction Combined with Residual BiLSTM and Sentence Bag Attention[J]. Computer Engineering, 2022, 48(10): 110-115,122.

江旭, 钱雪忠, 宋威. 结合残差BiLSTM与句袋注意力的远程监督关系抽取[J]. 计算机工程, 2022, 48(10): 110-115,122.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0062891

http://www.ecice06.com/EN/Y2022/V48/I10/110

Figures/Tables 8

References

[1] HAN S Q, HAO X L, HUANG H L.An event-extraction approach for business analysis from online Chinese news[J].Electronic Commerce Research and Applications, 2018, 28:244-260.
[2] FENG J, HUANG M L, ZHAO L, et al.Reinforcement learning for relation classification from noisy data[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence.Palo Alto, USA:AAAI Press, 2018:1-9.
[3] ZELENKO D, AONE C, RICHARDELLA A.Kernel methods for relation extraction[J].Journal of Machine Learning Research, 2020, 3(3):1083-1106.
[4] 孙紫阳, 顾君忠, 杨静.基于深度学习的中文实体关系抽取方法[J].计算机工程, 2018, 44(9):164-170. SUN Z Y, GU J Z, YANG J.Chinese entity relation extraction method based on deep learning[J].Computer Engineering, 2018, 44(9):164-170.(in Chinese)
[5] 鄂海红, 张文静, 肖思琪, 等.深度学习实体关系抽取研究综述[J].软件学报, 2019, 30(6):1793-1818. E H H, ZHANG W J, XIAO S Q, et al.Survey of entity relationship extraction based on deep learning[J].Journal of Software, 2019, 30(6):1793-1818.(in Chinese)
[6] BUNESCUR C, MOONEYR J.A shortest path dependency kernel for relation extraction[C]//Proceedings of Conference on Human Language Technology and Empirical Methods in Natural Language Processing.Philadelphia, USA:Association for Computational Linguistics, 2005:724-731.
[7] KAMBHATLA N.Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations[C]//Proceedings of the 43rd ACL International Conference.Philadelphia, USA:Association for the Computational Linguistics, 2017:178-181.
[8] UDDIN J, GHAZALI R, DERIS M M.Does number of clusters effect the purity and entropy of clustering[C]//Proceedings of 2016 International Conference on Soft Computing and Data Mining.Berlin, Germany:Springer, 2016:355-365.
[9] RUSSAKOVSKY O, DENG J, SU H, et al.ImageNet large scale visual recognition challenge[J].International Journal of Computer Vision, 2015, 115(3):211-252.
[10] CHEN C M, GUAN D J, SU Q K.Feature set identification for detecting suspicious URLs using Bayesian classification in social networks[J].Information Sciences, 2014, 289:133-147.
[11] ZHANG M, ZHANG J, SU J, et al.A composite kernel to extract relations between entities with both flat and structured features[C]//Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics.Philadelphia, USA:Association for Computational Linguistics, 2006:825-832.
[12] QIN P D, XU W R, WANG W Y.Robust distant supervision relation extraction via deep reinforcement learning[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.Philadelphia, USA:Association for Computational Linguistics, 2018:203-209.
[13] LI L F, NIE Y P, HAN W H, et al.A multi-attention-based bidirectional long short-term memory network for relation extraction[C]//Proceedings of International Conference on Neural Information Processing.Berlin, Germany:Springer, 2017:216-227.
[14] CASO A, ROSSI S.Users ranking in online social networks to support POI selection in small groups[C]//Proceedings of International Conference on Neural Information Processing.Berlin, Germany:Springer, 2020:104-107.
[15] GAO J, ZHOU T.Evaluating user reputation in online rating systems via an iterative group-based ranking method[J].Physica A:Statistical Mechanics and Its Applications, 2017, 473:546-560.
[16] PENG N Y, POON H, QUIRK C, et al.Cross-sentence N-ary relation extraction with graph LSTMs[J].Transactions of the Association for Computational Linguistics, 2017, 5:101-115.
[17] HUANG YI Y, WANG W Y.Deep residual learning for weakly supervised relation extraction[J].Journal of Machine Learning Research, 2019, 4(2):103-106.
[18] DIETTERICH T G, LATHROP R H, LOZANO-PÉREZ T.Solving the multiple instance problem with axis-parallel rectangles[J].Artificial Intelligence, 1997, 89(1/2):31-71.
[19] HAO Y C, ZHANG Y Z, LIU K, et al.An end-to-end model for question answering over knowledge base with cross-attention combining global knowledge[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.Philadelphia, USA:Association for Computational Linguistics, 2017:221-231.
[20] LIU H C, YOU J X, LI Z W, et al.Fuzzy Petri nets for knowledge representation and reasoning:a literature review[J].Engineering Applications of Artificial Intelligence, 2017, 60(C):45-56.
[21] LIN Y K, SHEN S Q, LIU Z Y, et al.Neural relation extraction with selective attention over instances[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.Philadelphia, USA:Association for Computational Linguistics, 2016:2124-2133.
[22] 刘正铭, 马宏, 刘树新, 等.一种融合节点文本属性信息的网络表示学习算法[J].计算机工程, 2018, 44(11):165-171. LIU Z M, MA H, LIU S X, et al.A network representation learning algorithm fusing with textual attribute information of nodes[J].Computer Engineering, 2018, 44(11):165-171.(in Chinese)
[23] LECUN Y, BENGIO Y, HINTON G.Deep learning[J].Nature, 2015, 521(7553):436-444.
[24] YE Z X, LING Z H.Distant supervision relation extraction with intra-bag and inter-bag attentions[EB/OL].[2021-09-11].https://arxiv.org/pdf/1904.00143.pdf.
[25] YAN X, MOU L L, LI G, et al.Classifying relations via long short term memory networks along shortest dependency path[EB/OL].[2021-09-11].https://arxiv.org/abs/1508.03720.
[26] KILINC O, UYSAL I.GAR:an efficient and scalable graph-based activity regularization for semi-supervised learning[J].Neurocomputing, 2018, 296:46-54.

Please choose a citation manager

Content to export