摘要：

多模态数据间交互式任务的出现对综合利用不同模态的知识提出了较高的要求，因此多模态知识图谱应运而生。在多模态知识图谱的构建过程中图像与文本实体是否指代同一对象尤为重要，这要求对中文跨模态实体进行实体对齐。针对该问题，提出一种基于多模态知识图谱的中文跨模态实体对齐方法。将图像信息引入实体对齐任务，面向领域细粒度图像和中文文本，设计单双流交互预训练语言模型(CCMEA)。基于自监督学习方法，利用视觉和文本编码器提取视觉和文本特征，并通过交叉编码器进行精细建模，最终采用对比学习方法计算图像和文本实体的匹配度。实验结果表明，在MUGE和Flickr30k-CN数据集上，CCMEA模型的平均召回率(MR)相比于Wukong_ViT-B基线模型分别提升了3.20和11.96个百分点，并在自建的TEXTILE数据集上MR达到94.3%。上述实验结果证明了该方法可以有效对齐中文跨模态实体，并且具有较高的准确性和实用性。

关键词: 多模态, 知识图谱, 实体对齐, 自监督, 纺织行业

Abstract:

Interactive tasks involving multi-modal data present advanced requirements for the comprehensive utilization of knowledge from different modalities, leading to the emergence of multi-modal knowledge graphs. When constructing these graphs, accurately determining whether image and text entities refer to the same object is particularly important for entity alignment of Chinese cross-modal entities. To address this problem, a Chinese cross-modal entity alignment method based on a multi-modal knowledge graph is proposed. Image information is introduced into the entity alignment task, and a single and dual-stream interactive pre-trained language model, namely CCMEA, is designed for domain-specific, fine-grained images and Chinese text. Utilizing a self-supervised learning method, Text-Visual features are extracted using Text-Visual Encoder, and fine-grained modeling is performed using cross-coders. Finally, a comparison learning method is employed to evaluate the degree of alignment between image and text entities. The experimental results show that the Mean Recall(MR) of the CCMEA model improved by 3.20 and 11.96 percentage points compared to that of the Wukong_ViT-B baseline model on the MUGE and Flickr30k-CN datasets, respectively. Furthermore, the model achieved a remarkable MR of 94.3% on the self-built TEXTILE dataset. These results demonstrate that the proposed method can effectively align Chinese cross-modal entities with high accuracy in practical applications.

Key words: multi-modal, knowledge graph, entity alignment, self-supervision, textile industry