Cross-Modal Named Entity Recognition Based on Semantic Fusion

doi:10.19678/j.issn.1000-3428.0069927

Abstract

Abstract:

To overcome the limitations of traditional single-modal Named Entity Recognition (NER) methods in processing multimodal data, this study proposes a novel cross-modal NER method based on semantic fusion. Unlike conventional approaches that use pretrained language models combined with Conditional Random Fields (CRFs), the proposed method introduces image information into the NER task and integrates the semantic content conveyed by images. First, the research status of NER tasks is reviewed, followed by a detailed explanation of the two core modules of the proposed cross-modal NER method: the image—text semantic fusion module and the bidirectional cross-modal matching module. Specifically, the image—text semantic fusion module generates textual representations based on the semantic content reflected in the images corresponding to the text, which are then combined with the original text information to form new text sequences. The bidirectional cross-modal matching module utilizes the newly generated text sequences to perform cross-modal matching with images. This approach of jointly considering image semantics and text before conducting cross-modal matching not only improves the accuracy of NER but also enriches the semantic representations of named entities. Finally, the effectiveness and superiority of the proposed method are demonstrated through comprehensive comparative and ablation experiments on standard benchmarks (Twitter-2015 and Twitter-2017). This study provides new insights and presents an effective methodological advancement for the NER of multimodal data.

Key words: natural language processing, Named Entity Recognition (NER), knowledge graph, multimodal, cross-modal attention mechanism

摘要：

针对传统单模态命名实体识别(NER)方法在处理多模态信息时的局限性, 提出一种基于语义联合的跨模态NER方法。不同于传统的预训练语言模型加条件随机场(CRF)的方法, 提出方法将图片信息引入NER任务, 并结合了图片所表达的语义。首先介绍了NER任务的研究现状; 然后详细说明了提出框架的图像文本语义联合模块和双向跨模态匹配模块, 其中, 图像文本语义联合模块根据文本对应图像所体现的语义生成文字表达, 并与原有的文本信息联合形成新的文本序列, 双向跨模态匹配模块利用新生成的文本序列与图像进行跨模态匹配, 这种联合图像语义和文本后进行跨模态匹配的方法不仅可以提高NER的准确性, 还能够丰富命名实体的语义表示; 最后在标准数据集(Twitter-2015、Twitter-2017)上进行对比实验和消融实验验证, 结果证明了提出方法在NER任务中的有效性和优越性, 为多模态数据中的NER任务提供了新的思路和方法。

关键词: 自然语言处理, 命名实体识别, 知识图谱, 多模态, 跨模态注意力机制

YANG Xinyi, HE Xiaohai, TENG Qizhi, CHEN Honggang, QING Linbo, WU Xiaohong. Cross-Modal Named Entity Recognition Based on Semantic Fusion[J]. Computer Engineering, 2026, 52(7): 345-353.

杨欣怡, 何小海, 滕奇志, 陈洪刚, 卿粼波, 吴晓红. 基于语义联合的跨模态命名实体识别[J]. 计算机工程, 2026, 52(7): 345-353.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0069927

https://www.ecice06.com/EN/Y2026/V52/I7/345

Figures/Tables 9

Fig.1 Overall structure of the proposed model framework

Fig.2 Composition of input on the text side

Fig.3 Final joint text — image feature representation which is obtained through the bidirectional cross-modal matching module

Fig.4 Example of text-image mismatching from Twitter-2015 dataset

References 30

1	LIU L P , WANG M L , ZHANG M Z , et al. UAMNer: uncertainty-aware multimodal named entity recognition in social media posts. Applied Intelligence, 2022, 52 (4): 4109- 4125. doi: 10.1007/s10489-021-02546-5
2	BEHMANESH S , TALEBPOUR A , SHAMSFARD M , et al. Improved relation span detection in question answering systems over extracted knowledge bases. Expert Systems with Applications, 2023, 224, 119973. doi: 10.1016/j.eswa.2023.119973
3	马恒志, 钱育蓉, 冷洪勇, 等. 知识图谱嵌入研究进展综述. 计算机工程, 2025, 51 (2): 18- 34. doi: 10.19678/j.issn.1000-3428.0068386
	MA H Z , QIAN Y R , LENG H Y , et al. Review of research progress on knowledge graph embedding. Computer Engineering, 2025, 51 (2): 18- 34. doi: 10.19678/j.issn.1000-3428.0068386
4	LI D Y , YAN L , ZHANG X W , et al. EventKGE: event knowledge graph embedding with event causal transfer. Knowledge-Based Systems, 2023, 278, 110917. doi: 10.1016/j.knosys.2023.110917
5	HUMPHREYS K, GAIZAUSKAS R, AZZAM S, et al. University of Sheffield: description of the LaSIE-Ⅱ system as used for MUC-7[C]//Proceedings of the 7th Message Understanding Conference. Washington D.C., USA: IEEE Press, 1995: 1-8.
6	LI J , SUN A X , HAN J L , et al. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering, 2022, 34 (1): 50- 70. doi: 10.1109/TKDE.2020.2981314
7	LAMPLE G, BALLESTEROS M, SUBRAMANIAN S, et al. Neural architectures for named entity recognition[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, USA: ACL Press, 2016: 260-270.
8	HUANG Z, XU W, YU K. Bidirectional LSTM-CRF models for sequence tagging[EB/OL]. [2024-05-20]. https://arxiv.org/abs/1508.01991.
9	唐卓然, 柳毅. 基于词汇融合和依存关系的中文命名实体识别. 计算机工程, 2024, 50 (10): 145- 153. doi: 10.19678/j.issn.1000-3428.0068226
	TANG Z R , LIU Y . Chinese named entity recognition based on lexicon fusion and dependency relation. Computer Engineering, 2024, 50 (10): 145- 153. doi: 10.19678/j.issn.1000-3428.0068226
10	SOUZA F, NOGUEIRA R, LOTUFO R. Portuguese named entity recognition using BERT-CRF[EB/OL]. [2024-05-20]. https://arxiv.org/abs/1909.10649.
11	MOON S, NEVES L, CARVALHO V. Multimodal named entity recognition for short social media posts[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Stroudsburg, USA: ACL Press, 2018: 1-9.
12	钟维幸, 王海荣, 王栋, 等. 多模态语义协同交互的图文联合命名实体识别方法. 广西科学, 2022, 29 (4): 681- 690.
	ZHONG W X , WANG H R , WANG D , et al. Image-text joint named entity recognition method based on multi-modal semantic interaction. Guangxi Sciences, 2022, 29 (4): 681- 690.
13	王海荣, 王彤, 徐玺, 等. CLGLF: 置信学习引导标签融合的多模态命名实体识别方法. 电子学报, 2024, 52 (7): 2429- 2437.
	WANG H R , WANG T , XU X , et al. CLGLF: confidence learning guides label fusion for multimodal named entity recognition method. Acta Electronica Sinica, 2024, 52 (7): 2429- 2437.
14	SANG E F T K, VEENSTRA J. Representing text chunks[C]//Proceedings of the 9th Conference on European Chapter of the Association for Computational Linguistics. Stroudsburg, USA: ACL Press, 1999: 173-179.
15	ZHANG P C, LI X J, HU X W, et al. VinVL: revisiting visual representations in vision-language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2021: 5579-5588.
16	SMITH R. An overview of the Tesseract OCR engine[C]// Proceedings of the 9th International Conference on Document Analysis and Recognition (ICDAR 2007). Washington D.C., USA: IEEE Press, 2007: 629-633.
17	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional Transformers for language understanding[C]//Proceedings of the 2019 Conference of the North America Chapter of the Association for Computational Linguistics. Stroudsburg, USA: ACL Press, 2019: 4171-4186.
18	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2016: 770-778.
19	LU D, NEVES L, CARVALHO V, et al. Visual attention model for name tagging in multimodal social media[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, USA: ACL Press, 2018: 1990-1999.
20	YE L W, ROCHAN M, LIU Z, et al. Cross-modal self-attention network for referring image segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2019: 10494-10503.
21	YU J F, JIANG J, YANG L, et al. Improving multimodal named entity recognition via entity span detection with unified multimodal Transformer[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, USA: ACL Press, 2020: 3342-3352.
22	LEHNERT W, CARDIE C, FISHER D, et al. University of Massachusetts: MUC-4 test results and analysis[C]// Proceedings of the 4th Conference on Message Understanding. Stroudsburg, USA: ACL Press, 1992: 151-158.
23	PASZKE A, GROSS S, MASSA F, et al. PyTorch: an imperative style, high-performance deep learning library[C]//Proceedings of the 33th International Conference on Neural Information Processing Systems. Stroudsburg, USA: ACL Press, 2019: 8026-8037.
24	HENDRYCKS D, GIMPEL K. Bridging nonlinearities and stochastic regularizers with Gaussian error linear units[EB/OL]. [2024-05-20]. https://doi.org/10.48550/arXiv.1606.08415.
25	MA X Z, HOVY E. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, USA: ACL Press, 2016: 1064-1074.
26	HAN K , XIAO A , WU E , et al. Transformer in Transformer. Advances in Neural Information Processing Systems, 2021, 34, 15908- 15919.
27	ZHANG Q, FU J L, LIU X Y, et al. Adaptive co-attention network for named entity recognition in tweets[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2018: 5674-5681.
28	WU Z W, ZHENG C M, CAI Y, et al. Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York, USA: ACM Press, 2020: 1038-1046.
29	XU B, HUANG S Z, SHA C F, et al. MAF: a general matching and alignment framework for multimodal named entity recognition[C]//Proceedings of the 15th ACM International Conference on Web Search and Data Mining. New York, USA: ACM Press, 2022: 1215-1223.
30	徐玺, 王海荣, 王彤, 等. 图文语义增强的多模态命名实体识别方法. 计算机应用研究, 2024, 41 (6): 1679- 1685.
	XU X , WANG H R , WANG T , et al. Textual-visual semantics-enhanced multimodal named entity recognition method. Application Research of Computers, 2024, 41 (6): 1679- 1685.

[1]	LIAO Guoan, XU Ji, CHEN Yanping. Semi-Supervised Text Classification Method Based on Multi-Granularity Semantic Margin Loss [J]. Computer Engineering, 2026, 52(7): 107-120.
[2]	DANG Weichao, PEI Lixian, GAO Gaimei, LIU Chunxia. Weakly-Supervised Temporal Action Localization via Video—Text Semantic Alignment and Full Video Dependency [J]. Computer Engineering, 2026, 52(7): 354-367.
[3]	LI Jinru, PAN Qingxian, GAO Zhaolong, WEI Kai, FAN Zequn. Crowdsourcing Task Recommendation Method Based on Collaborative Knowledge Graph and Hybrid Neural Network [J]. Computer Engineering, 2026, 52(7): 155-167.
[4]	PAN Keyue, GUO Wei, CHENG Xiang, LIU Yi. Extraction of Typical Scene Elements from Street View Images Based on Large Multimodal Models [J]. Computer Engineering, 2026, 52(7): 331-344.
[5]	WEI Fangda, LIU Miao, SUN Yi, WANG Jing, ZHAO Shenghui. Review of Audio and Video Deepfakes and Counterfeit Detection [J]. Computer Engineering, 2026, 52(7): 40-55.
[6]	LI Weigan, GUI Ning, QIAN Yurong, CHEN Jiaying. Hierarchical Network for Few-shot Multimodal Relation Extraction Based on Prompt-tuning [J]. Computer Engineering, 2026, 52(7): 368-376.
[7]	XUE Hanbing, NI Chen, LI Yuying, GUAN Jia, FANG Kai, CUI Wenqian. Entity Canonicalization Method Based on Dual-View Contrastive Learning and Submodular Optimization [J]. Computer Engineering, 2026, 52(6): 80-95.
[8]	WANG Yongqi, WANG Lei. Multimodal Sentiment Recognition Based on Cross-Modal Enhancement and Time-Step Gating [J]. Computer Engineering, 2026, 52(6): 258-267.
[9]	CUI Shuangxin, LU Bo, ZHANG Mingyue, ZHAO Yifan, WANG Ziming, LIU Xinyu, CHEN Chenglizhao. Multimodal Fusion-Based Method for 360° Image Quality and Aesthetic Assessment [J]. Computer Engineering, 2026, 52(6): 288-295.
[10]	YU Mengyuan, LIU Xiangyang. Ship Detection Method Based on Multimodal Visible and Infrared Image Fusion [J]. Computer Engineering, 2026, 52(6): 278-287.
[11]	XU Minchen, QU Dan, SI Nianwen, PENG Sisi, CHEN Yaqi. Technologies for Detecting Disinformation in Social Media: A Comprehensive Review [J]. Computer Engineering, 2026, 52(5): 60-80.
[12]	WANG Shuo, LI Ke, LI Zelin. Entity Alignment Optimization Method for Multiple Knowledge Graphs Fusion [J]. Computer Engineering, 2026, 52(5): 139-149.
[13]	LI Zeming, WANG Shuliang, SHANG Zihe, SHENG Ming. Review of Document Q & A Driven by Multimodal Retrieval-Augmented Generation (Invited) [J]. Computer Engineering, 2026, 52(4): 1-21.
[14]	HE Hongguang, XIAN Yantuan, XIANG Yan. Commonsense Knowledge Graph Completion Method Based on Relation-Constrained Contrastive Learning [J]. Computer Engineering, 2026, 52(4): 122-130.
[15]	ZHANG Tianzhi, ZHOU Gang, ZHANG Shuang, CHEN Jing, HUANG Ningbo, WU Hao. Target-Entity Sentiment Classification with Image-Text Multimodal Entity Alignment [J]. Computer Engineering, 2026, 52(3): 222-233.

Please choose a citation manager

Content to export