基于注意力和多模态混合融合的谣言检测方法

doi:10.19678/j.issn.1000-3428.0059683

摘要/Abstract

摘要： 社交媒体内容结构复杂，很多虚假信息掺杂在真实内容中或者真实图片配以杜撰的文字内容，基于单个模态难以有效检测谣言。为了有效解决这一问题，提出了基于注意力机制与Dempster组合规则的混合融合方法（DHF）。首先，新增用户模态，并提取文本、视觉和用户三个模态的特征向量。其次，利用注意力机制，对词语和视觉进行双向匹配，给予对谣言检测具有更多贡献的词语和视觉神经元更大的权值。最后，在前期融合和后期融合均加入注意力机制，实现特征和决策的自动加权，并使用Dempster组合规则实现混合融合。DHF在真实的中文Weibo数据集和外文Twitter数据集上分别达到了97.44%和92.35%的准确率，Accuracy和F1-score指标均超过了多个基准方法和两个先进的多模态方法。

Abstract: Owing to the complexity of content structure on social media, there are many rumors mixed in the real content. Similarly, there are many real pictures with fabricated content. So, it is difficult to detect rumors effectively based on single modal. In order to solve this problem, a hybrid fusion method based on attention mechanism and Dempster combination rules (DHF) was introduced. Firstly, the user modal was integrated, and the feature vectors of text modal, vision modal and user modal were extracted. Then, the attention mechanism was utilized to give more weight to words and visual neurons that contributed more to rumor detection. At last, the attention mechanism was added to early fusion and late fusion to achieve the automatic weighting of the features and decisions. The hybrid fusion of early fusion and late fusion was implemented by Dempster combination rules. The DHF achieves 97.44% and 92.35% accuracy on real Chinese Weibo data set and foreign Twitter data set respectively. The accuracy and F1 score are both better than the base-line methods and two advanced multimodal methods.

陶霄 , 朱焱 , 李春平. 基于注意力和多模态混合融合的谣言检测方法[J]. 计算机工程, doi: 10.19678/j.issn.1000-3428.0059683.

Tao Xiao , ZHU Yan , LI Chunping. Rumor Detection with Attention and Multi-modal Hybrid Fusion[J]. Computer Engineering, doi: 10.19678/j.issn.1000-3428.0059683.

参考文献

[1] CASTILLO C, MENDOZA M, POBLETE B. Information credibility on Twitter[C]// Proceedings of the 20th International Conference on World Wide Web. Hyderabad: IEEE, 2011: 675-684.
[2] MA J, GAO W, MITRA P, et al. Detecting rumors from microblogs with recurrent neural networks[C]// Proceedings of the 25th International Joint Conference on Artiﬁcial Intelligence. New York: ACM, 2016: 3818–3824.
[3] Bhatt G, Sharma A, Sharma S, et al. Combining Neural, Statistical and External Features for Fake News Identification[C]// Companion Proceedings of The International Conference of World Wide Web 2018. Lyon, France: ACM, 2018: 1353-1357.
[4] ZHIYUAN Z, YUAN Q, ZHENG L, et al. A C-GRU Neural Network for Rumors Detection [C]// The 5th IEEE International Conference on Cloud Computing and Intelligence Systems. Nanjing, China: IEEE, 2018: 704-708.
[5] Qi P, Cao J, Yang T, et al. Exploiting Multi-domain Visual Information for Fake News Detection[C]// 2019 IEEE International Conference on Data Mining. Beijing: IEEE, 2019: 518-527.
[6] Bahdanau D, Cho K, Bengio Y. Neural Machine Translation by Jointly Learning to Align and Translate[C]// 2015 International Conference on Learning Representations. San Diego, CA: ACM, 2015: 1104-1116.
[7] VASWANI A, SHAZEER N, et al. Attention Is All You Need[C]// Advances in Neural Information Processing Systems 30. Long Beach, CA, USA:NIPS, 2017: 5998-6008.
[8] HE Jun, ZHANG Caiqing, LI Xiaozhen, et al. Survey of Research on Multimodal Fusion Technology for Deep Learning[J]. Computer Engineering, 2020, 46(5), 1-11. (in Chinese) 何俊, 张彩庆, 李小珍, 等. 面向深度学习的多模态融合技术研究综述[J]. 计算机工程, 2020, 46(5), 1-11.
[9] SUN Yingying, JIA Zhentang, ZHU Haoyu. Survey of Multimodal Deep Learning [J]. Computer Engineering and Applications. 2020, 56(21), 1-10. (in Chinese) 孙影影, 贾振堂, 朱昊宇. 多模态深度学习综述[J]. 计算机工程与应用. 2020, 56(21), 1-10.
[10] JIN Z W, CAO J, GUO H, et al. Multimodal Fusion with Recurrent Neural Networks for Rumor Detection on Microblogs[C]// Proceedings of the 25th ACM international conference on Multimedia. Mountain View, California, USA: ACM, 2017: 795-816.
[11] Devamanyu H, Sruthi G, et al. Self-Attentive Feature-Level Fusion for Multimodal Emotion Detection[C]// Proceedings of International Conference on Multimedia Information Processing and Retrieval 2018. Miami, FL, USA: ACM, 2018: 196-201.
[12] LU Liangfeng, XIE Zhijun, YE Hongwu. Object Recognition Algorithm Based on RGB Feature and Depth Feature Fusing[J]. Computer Engineering, 2016, 42(5), 186-193. (in Chinese) 卢良锋,谢志军,叶宏武. 基于RGB特征与深度特征融合的物体识别算法[J]. 计算机工程, 2016, 42(5), 186-193.
[13] Geng Y., Lin Z., Fu P., Wang W. Rumor Detection on Social Media: A Multi-view Model Using Self-attention Mechanism[C]// Proceedings of International Conference on Computational Science. Berlin: Springer, Cham:ACM, 2019: 339-352.
[14] PENNINGTON J, SOCHER R, CHRISTOPHER D. GloVe: Global Vectors for Word Representation[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Qatar: Association for Computational Linguistics, 2014: 1532–1543.
[15] DEVLIN J, CHANG M W, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[C]// Proceedings of the 2019 Conference of the NAC. Minnesota: Association for Computational Linguistics, 2019: 4171–4186.
[16] MIKOLOV T, CHEN K, et al. Efficient Estimation of Word Representations in Vector Space[C]// Proceedings of the International Conference on Learning Representations. Arizona: ACM, 2013:1-8.
[17] Dempster A P. A generalization of Bayesian inference[J]. Journal of the Royal Statal Society, 1968, 30(2): 205-232.
[18] WANG Y Q, MA F L, et al. EANN: Event Adversarial Neural Networks for Multi-Modal Fake News Detection[C]// Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. London: ACM, 2018: 849-857.
[19] Alex G, Greg W, Ivo D. Neural Turing Machines[J]. arXiv preprint arXiv:1410.6247, 2014.
[20] Thang L, Hieu P, Christopher D. Effective Approaches to Attention-based Neural Machine Translation[C]// Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal: EMNLP, 2015: 1412-1421.