基于知识图谱增强的领域多模态实体识别

doi:10.19678/j.issn.1000-3428.0068225

摘要/Abstract

摘要： 针对特定领域中文命名实体识别存在句子简短或歧义等问题，提出了一种利用学科图谱和图像提高实体识别准确率的模型，旨在利用领域图谱和图像，提高计算机学科领域短文本中实体识别的准确率。使用基于BERT-BiLSTM的模型提取文本特征，使用ResNet152提取图像特征，并使用分词工具获得句子中的名词实体。通过BERT将名词实体与图谱节点进行特征嵌入，使用余弦相似度查找句子中的分词在学科图谱中最相似的节点，保留此节点领域为1的邻居节点，生成最佳匹配子图，作为句子的语义补充。模型使用多层感知机将文本、图像和子图三种特征映射到同一空间，并通过独特的门结构实现文本和图像的细粒度跨模态特征融合。最后通过交叉注意力机制将多模态特征与子图特征进行融合，输入解码器进行实体标记。在Twitter2015、Twitter2017和自建计算机学科数据集上同基线模型进行了比较，所提方法在领域数据集上的精确率、召回率和F1值分别可达88.56%、87.47%和88.01%，表明利用领域知识图谱能提升实体识别效果。

Abstract: In response to challenges encountered in Chinese Named Entity Recognition (NER) within specific domains, such as short sentences and ambiguity, a model is proposed to enhance entity recognition accuracy by utilizing domain-specific knowledge graphs and images. The aim is to leverage domain graphs and images to improve entity recognition in short texts related to computer science. The model employs a BERT-BiLSTM-based architecture to extract textual features, a ResNet152-based approach to extract image features, and a word segmentation tool to obtain noun entities from sentences. These noun entities are then embedded with knowledge graph nodes using BERT. The model uses cosine similarity to find the most similar nodes in the knowledge graph for the segmented words in the sentence. It retains neighboring nodes with domain label 1 to generate an optimal matching subgraph for semantic enrichment of the sentence. A multi-layer perceptron (MLP) is employed to map the textual, image, and subgraph features into the same space. A unique gating mechanism is utilized to achieve fine-grained cross-modal feature fusion between textual and image features. Finally, a cross-attention mechanism is applied to fuse multimodal features with subgraph features, which are then fed into the decoder for entity labeling. Experimental comparisons with relevant baseline models were conducted on Twitter 2015, Twitter 2017, and a self-constructed computer science dataset. Results indicate that the proposed approach achieves precision, recall, and F1-score of 88.56%, 87.47%, and 88.01% on the domain dataset, demonstrating the effectiveness of incorporating domain knowledge graphs for entity recognition.

李华昱, 张智康, 闫阳, 岳阳. 基于知识图谱增强的领域多模态实体识别[J]. 计算机工程, doi: 10.19678/j.issn.1000-3428.0068225.

LI Huayu, ZHANG Zhikang, YAN Yang, YUE Yang. Enhanced Domain Multi-modal Entity Recognition based on Knowledge Graph[J]. Computer Engineering, doi: 10.19678/j.issn.1000-3428.0068225.

参考文献

[1] 潘梦竹, 李千目, 邱天. 深度多模态表示学习的研究综述[J]. 计算机工程与应用, 2023, 59(02): 48-64. PAN M Z, LI Q M, QIU T. Survey of research on deep multimodal representation learning[J]. Computer Engi- neering and Applications, 2023, 59(02): 48-64.
[2] 王颖洁, 张程烨, 白凤波, 等. 中文命名实体识别研究综述[J]. 计算机科学与探索, 2023, 17(2): 324-341. WANG Y J, ZHANG C Y, BAI F B, et al. Review of Chinese named entity recognition research[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(2): 324-341.
[3] 隋国华, 李陶然, 刘昊, 等. 基于图表示学习的领域知识图谱推理技术研究[J]. 计算机工程, 2023, 49(9): 89-98. SUI G H, LI T R, LIU H, et al. Research on domain knowledge graph inference technology based on graph representation learning[J]. Computer Engineering, 2023, 49(9): 89-98.
[4] 潘正高. 基于规则和统计相结合的中文命名实体识别研究[J]. 情报科学, 2012, 30(5): 708-712. PAN Z G. Research on the recognition of Chinese named entity based on rules and statistics[J]. Information Sci- ence, 2012, 30(5): 708-712.
[5] 闫萍. 基于规则和概率统计相结合的中文命名实体识别研究[J]. 计算机与数字工程, 2011, 39(9): 88-91. YAN P. Research on the identification for Chinese named entity based on combination of rules and statistic analysis[J]. Computer & Digital Engineering, 2011, 39(9): 88-91.
[6] 王欢, 朱文球, 吴岳忠, 等. 基于数控机床设备故障领域的命名实体识别[J]. 工程科学学报, 2020, 42(4): 476-482. WANG H, ZHU W Q, WU Y Z, et al. Named entity recognition based on equipment and fault field of CNC machine tools[J]. Chinese Journal of Engineering, 2020, 42(4): 476-482.
[7] 杨培, 杨志豪, 罗凌, 等. 基于注意机制的化学药物命名实体识别[J]. 计算机研究与发展, 2018, 55(7): 1548-1556. YANG P, YANG Z H, LUO L, et al. An attention-based approach for chemical compound and drug named entity recognition[J]. Journal of Computer Research and De- velopment, 2018, 55(7): 1548-1556.
[8] ZHAO Z, YANG Z, LUO L, et al. Disease named entity recognition from biomedical literature using a novelconvolutional neural network[J]. BMC medical ge- nomics, 2017, 10: 75-83.
[9] 王蓬辉, 李明正, 李思. 基于数据增强的中文医疗命名实体识别[J]. 北京邮电大学学报, 2020, 43(5): 84-90. WANG P H, LI M Z, LI S. Data augmentation for Chi- nese clinical named entity recognition[J]. Journal of Bei- jing University of Posts and Telecommunications, 2020, 43(5): 84-90.
[10] AGUILAR G, MAHARJAN S, LÓPEZ-MONROY A P, et al. A multi-task approach for named entity recognition in social media data[J]. arXiv preprint arXiv: 1906.04135, 2019.
[11] MOON S, NEVES L, CARVALHO V. Multimodal named entity recognition for short social media posts[J]. arXiv preprint arXiv:1802.07862, 2018.
[12] ZHENG C, WU Z, WANG T, et al. Object-aware multi- modal named entity recognition in social media posts with adversarial learning[J]. IEEE Transactions on Mul- timedia, 2020, 23: 2520-2532.
[13] YU J, JIANG J, YANG L, et al. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer[C]. Association for Computational Linguistics, 2020.
[14] CHEN D, LI Z, GU B, et al. Multimodal named entity recognition with image attributes and image knowledge[C]//Database Systems for Advanced Appli- cations: 26th International Conference, DASFAA 2021, Taipei, Taiwan, April 11–14, 2021, Proceedings, Part II 26. Springer International Publishing, 2021: 186-201.
[15] ZHAO F, LI C, WU Z, et al. Learning from Different text-image Pairs: A Relation-enhanced Graph Convolu- tional Network for Multimodal NER[C]//Proceedings of the 30th ACM International Conference on Multimedia. 2022: 3983-3992.
[16] WANG J, YANG Y, LIU K, et al. M3S: Scene graph driven multi-granularity multi-task learning for mul- ti-modal NER[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022, 31: 111-120. [17] 李家瑞, 李华昱, 闫阳. 面向多源异质数据源的学科知识图谱构建方法[J]. 计算机系统应用, 2021, 30(10): 59-67. LI J R, LI H Y, YAN Y. Construction of discipline knowledge graph for multi-source heterogeneous data sources[J]. Computer Systems & Applications, 2021, 30(10): 59-67.
[18] SANG E F, VEENSTRA J. Representing text chunks[J]. arXiv preprint cs/9907006, 1999.
[19] DENG J, DONG W, SOCHER R, et al. ImageNet: A large-scale hierarchical image database[C]//2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009: 248-255.
[20] ZHOU B, ZHANG Y, SONG K, et al. A span-based mul- timodal variational autoencoder for semi-supervised multimodal named entity recognition[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022: 6293-6302.
[21] 胡新棒, 于溆乔, 李邵梅, 张建朋. 基于知识增强的中文命名实体识别[J]. 计算机工程, 2021, 47(11): 84-92. HU X B, YU X Q, LI S M, ZHANG J P. Chinese named entity recognition based on knowledge enhancement[J]. Computer Engineering, 2021, 47(11): 84-92.
[22] ZHANG D, WEI S, LI S, et al. Multi-modal graph fusion for named entity recognition with targeted visual guid- ance[C]//Proceedings of the AAAI conference on artifi- cial intelligence. 2021, 35(16): 14347-14355.
[23] ZHANG Q, FU J, LIU X, et al. Adaptive co-attention network for named entity recognition in tweets[C]//Proceedings of the AAAI conference on arti- ficial intelligence. 2018, 32(1).
[24] LU D, NEVES L, CARVALHO V, et al. Visual attention model for name tagging in multimodal social me- dia[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). 2018: 1990-1999.
[25] DEVLIN J, CHANG M W, LEE K, et al. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding[J]. arXiv preprint arXiv: 1810.04805, 2018.
[26] HU S, ZHANG H, HU X, et al. Chinese named entity recognition based on BERT-CRF model[C]//2022 IEEE/ACIS 22nd International Conference on Computer and Information Science (ICIS). IEEE, 2022: 105-108.
[27] XU B, HUANG S, SHA C, et al. MAF: a general matching and alignment framework for multimodal named entity recognition[C]//Proceedings of the fif- teenth ACM international conference on web search and data mining. 2022: 1215-1223.
[28] JIA M, SHEN L, SHEN X, et al. Mner-qg: An end-to-end mrc framework for multimodal named entity recognition with query grounding[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(7): 8032-8040.

选择文件类型/文献管理软件名称

选择包含的内容