基于语义联合的跨模态命名实体识别

doi:10.19678/j.issn.1000-3428.0069927

摘要/Abstract

摘要： 针对传统单模态命名实体识别(NER)方法在处理多模态信息时的局限性,提出一种基于语义联合的跨模态NER方法。不同于传统的预训练语言模型加条件随机场(CRF)的方法,提出方法将图片信息引入NER任务,并结合了图片所表达的语义。首先介绍了NER任务的研究现状;然后详细说明了提出框架的图像文本语义联合模块和双向跨模态匹配模块,其中,图像文本语义联合模块根据文本对应图像所体现的语义生成文字表达,并与原有的文本信息联合形成新的文本序列,双向跨模态匹配模块利用新生成的文本序列与图像进行跨模态匹配,这种联合图像语义和文本后进行跨模态匹配的方法不仅可以提高NER的准确性,还能够丰富命名实体的语义表示;最后在标准数据集(Twitter-2015、Twitter-2017)上进行对比实验和消融实验验证,结果证明了提出方法在NER任务中的有效性和优越性,为多模态数据中的NER任务提供了新的思路和方法。

关键词: 自然语言处理, 命名实体识别, 知识图谱, 多模态, 跨模态注意力机制

Abstract: To overcome the limitations of traditional single-modal Named Entity Recognition (NER) methods in processing multimodal data, this study proposes a novel cross-modal NER method based on semantic fusion. Unlike conventional approaches that use pretrained language models combined with Conditional Random Fields (CRFs), the proposed method introduces image information into the NER task and integrates the semantic content conveyed by images. First, the research status of NER tasks is reviewed, followed by a detailed explanation of the two core modules of the proposed cross-modal NER method: the image—text semantic fusion module and the bidirectional cross-modal matching module. Specifically, the image—text semantic fusion module generates textual representations based on the semantic content reflected in the images corresponding to the text, which are then combined with the original text information to form new text sequences. The bidirectional cross-modal matching module utilizes the newly generated text sequences to perform cross-modal matching with images. This approach of jointly considering image semantics and text before conducting cross-modal matching not only improves the accuracy of NER but also enriches the semantic representations of named entities. Finally, the effectiveness and superiority of the proposed method are demonstrated through comprehensive comparative and ablation experiments on standard benchmarks (Twitter-2015 and Twitter-2017). This study provides new insights and presents an effective methodological advancement for the NER of multimodal data.

Key words: natural language processing, Named Entity Recognition (NER), knowledge graph, multimodal, cross-modal attention mechanism

中图分类号:

TP393

杨欣怡, 何小海, 滕奇志, 陈洪刚, 卿粼波, 吴晓红. 基于语义联合的跨模态命名实体识别[J]. 计算机工程, 2026, 52(7): 345-353.

YANG Xinyi, HE Xiaohai, TENG Qizhi, CHEN Honggang, QING Linbo, WU Xiaohong. Cross-Modal Named Entity Recognition Based on Semantic Fusion[J]. Computer Engineering, 2026, 52(7): 345-353.

https://www.ecice06.com/CN/Y2026/V52/I7/345

参考文献

[1] LIU L P, WANG M L, ZHANG M Z, et al. UAMNer: uncertainty-aware multimodal named entity recognition in social media posts[J]. Applied Intelligence, 2022, 52(4): 4109-4125.
[2] BEHMANESH S, TALEBPOUR A, SHAMSFARD M, et al. Improved relation span detection in question answering systems over extracted knowledge bases[J]. Expert Systems with Applications, 2023, 224: 119973.
[3] 马恒志, 钱育蓉, 冷洪勇, 等. 知识图谱嵌入研究进展综述[J]. 计算机工程, 2025, 51(2): 18-34. MA H Z, QIAN Y R, LENG H Y, et al. Review of research progress on knowledge graph embedding[J]. Computer Engineering, 2025, 51(2): 18-34. (in Chinese)
[4] LI D Y, YAN L, ZHANG X W, et al. EventKGE: event knowledge graph embedding with event causal transfer[J]. Knowledge-Based Systems, 2023, 278: 110917.
[5] HUMPHREYS K, GAIZAUSKAS R, AZZAM S, et al. University of Sheffield: description of the LaSIE-II system as used for MUC-7[C]//Proceedings of the 7th Message Understanding Conference. Washington D.C., USA: IEEE Press, 1995: 1-8.
[6] LI J, SUN A X, HAN J L, et al. A survey on deep learning for named entity recognition[J]. IEEE Transactions on Knowledge and Data Engineering, 2022, 34(1): 50-70.
[7] LAMPLE G, BALLESTEROS M, SUBRAMANIAN S, et al. Neural architectures for named entity recognition[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, USA: ACL Press, 2016: 260-270.
[8] HUANG Z, XU W, YU K. Bidirectional LSTM-CRF models for sequence tagging[EB/OL].[2024-05-20]. https://arxiv.org/abs/1508.01991.
[9] 唐卓然, 柳毅. 基于词汇融合和依存关系的中文命名实体识别[J]. 计算机工程, 2024, 50(10): 145-153. TANG Z R, LIU Y. Chinese named entity recognition based on lexicon fusion and dependency relation[J]. Computer Engineering, 2024, 50(10): 145-153. (in Chinese)
[10] SOUZA F, NOGUEIRA R, LOTUFO R. Portuguese named entity recognition using BERT-CRF[EB/OL].[2024-05-20]. https://arxiv.org/abs/1909.10649.
[11] MOON S, NEVES L, CARVALHO V. Multimodal named entity recognition for short social media posts[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Stroudsburg, USA: ACL Press, 2018: 1-9.
[12] 钟维幸, 王海荣, 王栋, 等. 多模态语义协同交互的图文联合命名实体识别方法[J]. 广西科学, 2022, 29(4): 681-690. ZHONG W X, WANG H R, WANG D, et al. Image-text joint named entity recognition method based on multi-modal semantic interaction[J]. Guangxi Sciences, 2022, 29(4): 681-690. (in Chinese)
[13] 王海荣, 王彤, 徐玺, 等. CLGLF: 置信学习引导标签融合的多模态命名实体识别方法[J]. 电子学报, 2024, 52(7): 2429-2437. WANG H R, WANG T, XU X, et al. CLGLF: confidence learning guides label fusion for multimodal named entity recognition method[J]. Acta Electronica Sinica, 2024, 52(7): 2429-2437. (in Chinese)
[14] SANG E F T K, VEENSTRA J. Representing text chunks[C]//Proceedings of the 9th Conference on European Chapter of the Association for Computational Linguistics. Stroudsburg, USA: ACL Press, 1999: 173-179.
[15] ZHANG P C, LI X J, HU X W, et al. VinVL: revisiting visual representations in vision-language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2021: 5579-5588.
[16] SMITH R. An overview of the Tesseract OCR engine[C]//Proceedings of the 9th International Conference on Document Analysis and Recognition (ICDAR 2007). Washington D.C., USA: IEEE Press, 2007: 629-633.
[17] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional Transformers for language understanding[C]//Proceedings of the 2019 Conference of the North America Chapter of the Association for Computational Linguistics. Stroudsburg, USA: ACL Press, 2019:4171-4186.
[18] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2016: 770-778.
[19] LU D, NEVES L, CARVALHO V, et al. Visual attention model for name tagging in multimodal social media[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, USA: ACL Press, 2018: 1990-1999.
[20] YE L W, ROCHAN M, LIU Z, et al. Cross-modal self-attention network for referring image segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2019: 10494-10503.
[21] YU J F, JIANG J, YANG L, et al. Improving multimodal named entity recognition via entity span detection with unified multimodal Transformer[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, USA: ACL Press, 2020: 3342-3352.
[22] LEHNERT W, CARDIE C, FISHER D, et al. University of Massachusetts: MUC-4 test results and analysis[C]//Proceedings of the 4th Conference on Message Understanding. Stroudsburg, USA: ACL Press, 1992: 151-158.
[23] PASZKE A, GROSS S, MASSA F, et al. PyTorch: an imperative style, high-performance deep learning library[C]//Proceedings of the 33th International Conference on Neural Information Processing Systems. Stroudsburg, USA: ACL Press, 2019: 8026-8037.
[24] HENDRYCKS D, GIMPEL K. Bridging nonlinearities and stochastic regularizers with Gaussian error linear units[EB/OL].[2024-05-20]. https://doi.org/10.48550/arXiv.1606.08415.
[25] MA X Z, HOVY E. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, USA: ACL Press, 2016: 1064-1074.
[26] HAN K, XIAO A, WU E, et al. Transformer in Transformer[J]. Advances in Neural Information Processing Systems, 2021, 34: 15908-15919.
[27] ZHANG Q, FU J L, LIU X Y, et al. Adaptive co-attention network for named entity recognition in tweets[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2018: 5674-5681.
[28] WU Z W, ZHENG C M, CAI Y, et al. Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York, USA: ACM Press, 2020: 1038-1046.
[29] XU B, HUANG S Z, SHA C F, et al. MAF: a general matching and alignment framework for multimodal named entity recognition[C]//Proceedings of the 15th ACM International Conference on Web Search and Data Mining. New York, USA: ACM Press, 2022: 1215-1223.
[30] 徐玺, 王海荣, 王彤, 等. 图文语义增强的多模态命名实体识别方法[J]. 计算机应用研究, 2024, 41(6): 1679-1685. XU X, WANG H R, WANG T, et al. Textual-visual semantics-enhanced multimodal named entity recognition method[J]. Application Research of Computers, 2024, 41(6): 1679-1685. (in Chinese)

选择文件类型/文献管理软件名称

选择包含的内容