基于多任务学习的多模态命名实体识别方法

doi:10.19678/j.issn.1000-3428.0064087

摘要/Abstract

摘要： 针对传统多模态命名实体识别方法无法有效融合图文模态信息且不能区分易混淆实体等问题，提出一种基于多任务学习的多模态命名实体识别方法，通过对比融合辅助任务促进图文模态信息的融合，通过实体聚类辅助任务提升模型对易混淆实体的判断能力。利用BERT预训练语言模型和ResNet模型分别对原始文本和图片进行特征映射获得相应的特征向量，并利用跨模态Transformer结构融合图文模态信息。在多模态命名实体识别任务基础上，增加对比融合辅助任务促进图文模态信息融合，增加实体聚类辅助任务学习实体类别之间的差异，提升模型对易混淆实体的区分能力。最后，利用条件随机场层学习上下文转移概率，并输出最优预测结果。实验结果显示，在国际公开数据集Twitter-2017上，所提方法相较于基线方法取得了更高的准确率、召回率和F1值，其中F1值可达85.59%，表明对比融合辅助任务和实体聚类辅助任务能够促进模型对实体的识别效果。

关键词: 命名实体识别, 多任务学习, 多模态信息, 对比学习, 聚类

Abstract: With the aim of overcoming the ineffectiveness of traditional multi-modal Named Entity Recognition (NER)methods in integrating text and image modal information and distinguishing confusable entities, a multi-modal NER method based on multi-task learning is proposed.Here, the fusion of modal information is promoted by a contrast fusion auxiliary task, and the ability to differentiate confusable entities is improved by an entity clustering auxiliary task.First, BERT pre-trained language modal and ResNet model are used to obtain feature vectors, and a cross-modal Transformer is used to fuse text and image modal information.Second, based on the multi-modal NER task, a contrast fusion auxiliary task is added to promote the fusion of image and text modal information.An entity clustering auxiliary task is added to learn the differences between entity categories and improve the ability of the model to distinguish easily confusable entities.Finally, a Conditional Random Field(CRF) layer is used to learn the context transition probability and output optimal prediction results.Experimental results show that, on the international open dataset Twitter-2017, the proposed method achieved higher accuracies, recall rates, and F1-scores than baseline methods, with an F1-score of up to 85.59% being attained.The results show that the added contrast fusion and entity clustering auxiliary tasks improve the recognition effectiveness of the model.

Key words: Named Entity Recognition(NER), multi-task learning, multi-modal information, contrastive learning, clustering

中图分类号:

TP18

李晓腾, 张盼盼, 勾智楠, 高凯. 基于多任务学习的多模态命名实体识别方法[J]. 计算机工程, 2023, 49(4): 114-119.

LI Xiaoteng, ZHANG Panpan, GOU Zhinan, GAO Kai. Multi-Modal Named Entity Recognition Method Based on Multi-Task Learning[J]. Computer Engineering, 2023, 49(4): 114-119.

https://www.ecice06.com/CN/Y2023/V49/I4/114

图/表 6

20230417183433

20230417183437

20230417183441

20230417183444

20230417183447

20230417183451

参考文献

[1] LI J, SUN A X, HAN J L, et al.A survey on deep learning for named entity recognition[J].IEEE Transactions on Knowledge and Data Engineering, 2022, 34(1):50-70.
[2] MOON S, NEVES L, CARVALHO V.Multimodal named entity recognition for short social media posts[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologie.Stroudsburg, USA:Association for Computational Linguistics, 2018:852-860.
[3] ZHANG Q, FU J, LIU X, et al.Adaptive co-attention network for named entity recognition in Tweets[C]//Proceedings of the 32th AAAI Conference on Artificial Intelligence.Palo Alto, USA:AAAI Press, 2018:5674-5681.
[4] LU D, NEVES L, CARVALHO V, et al.Visual attention model for name tagging in multimodal social media[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.Stroudsburg, USA:Association for Computational Linguistics, 2018:1990-1999.
[5] HE K M, ZHANG X Y, REN S Q, et al.Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:770-778.
[6] YU J F, JIANG J, YANG L, et al.Improving multimodal named entity recognition via entity span detection with unified multimodal transformer[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Stroudsburg, USA:Association for Computational Linguistics, 2020:3342-3352.
[7] DEVLIN J, CHANG M W, LEE K, et al.BERT:pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.Stroudsburg, USA:Association for Computational Linguistics, 2019:4171-4186.
[8] CIPOLLA R, GAL Y, KENDALL A.Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:7482-7491.
[9] LI Y J, CARAGEA C.Multi-task stance detection with sentiment and stance lexicons[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing.Stroudsburg, USA:Association for Computational Linguistics, 2019:6298-6304.
[10] CLARK K, LUONG M T, KHANDELWAL U, et al.BAM! Born-again multi-task networks for natural language understanding[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Stroudsburg, USA:Association for Computational Linguistics, 2019:5931-5937.
[11] WANG X, LYU J G, DONG L, et al.Multitask learning for biomedical named entity recognition with cross-sharing structure[J].BMC Bioinformatics, 2019, 20(1):427.
[12] ZHANG Y, YANG Q.An overview of multi-task learning[J].National Science Review, 2018, 5(1):30-43.
[13] CHEN Z, BADRINARAYANAN V, LEE C Y, et al.GradNorm:gradient normalization for adaptive loss balancing in deep multitask networks[C]//Proceedings of the 35th International Conference on Machine Learning.Stockholm, Sweden:PMLR, 2018:794-803.
[14] YANG Y, HOSPEDALES T M.Deep multi-task representation learning:a tensor factorisation approach[C]//Proceedings of the 5th International Conference on Learning Representations.Toulon, France:[s.n.], 2017:1-14.
[15] REI M.Semi-supervised multitask learning for sequence labeling[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.Stroudsburg, USA:Association for Computational Linguistics, 2017:2121-2130.
[16] LIN Y, YANG S Q, STOYANOV V, et al.A multi-lingual multi-task architecture for low-resource sequence labeling[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.Stroudsburg, USA:Association for Computational Linguistics, 2018:799-809.
[17] VASWANI A, SHAZEER N, PARMAR N, et al.Attention is all you need[C]//Proceedings of NIPS'17.Cambridge, USA:MIT Press, 2017:5998-6008.
[18] CHEN T, KORNBLITH S, NOROUZI M, et al.A simple framework for contrastive learning of visual representations[C]//Proceedings of the 37th International Conference on Machine Learning.New York, USA:ACM Press, 2020:1597-1607.
[19] FANG H C, WANG S C, ZHOU M, et al.CERT:contrastive self-supervised learning for language understanding[EB/OL].[2022-01-12].https://arxiv.org/abs/2005.12766.
[20] GIORGI J, NITSKI O, WANG B, et al.DeCLUTR:deep contrastive learning for unsupervised textual representations[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing.Stroudsburg, USA:Association for Computational Linguistics, 2021:879-895.
[21] WU Z F, WANG S N, GU J T, et al.CLEAR:contrastive learning for sentence representation[EB/OL].[2022-01-12].https://arxiv.org/abs/2012.15466.
[22] YOU Y N, CHEN T L, SHEN Y, et al.Graph contrastive learning automated[EB/OL].[2022-01-12].https://arxiv.org/abs/2106.07594.
[23] MA X Z, HOVY E.End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.Stroudsburg, USA:Association for Computational Linguistics, 2016:1064-1074.
[24] HUANG Z H, XU W, YU K.Bidirectional LSTM-CRF models for sequence tagging[EB/OL].[2022-01-12].https://arxiv.org/abs/1508.01991.
[25] LAMPLE G, BALLESTEROS M, SUBRAMANIAN S, et al.Neural architectures for named entity recognition[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.Stroudsburg, USA:Association for Computational Linguistics, 2016:260-270.

选择文件类型/文献管理软件名称

选择包含的内容