Rumor Detection Method Based on Attention and Multi-Modal Hybrid Fusion

doi:10.19678/j.issn.1000-3428.0059683

Abstract

Abstract: Content on social media is characterized by high structural complexity.Many rumors are mixed with real information, and real pictures are tagged with fabricated description. So it is difficult to detect rumors effectively based on single modal methods.In order to solve the problem, a method is proposed based on hybrid fusion of the attention mechanism and Dempster's rule of combination.The method adds three kinds of modal feature vectors, including text, vision and user.Then the attention mechanism is utilized to give more weight to words and visual neurons that contribute more to rumor detection, making bidirectional matching between words and vision.The attention mechanism is added to early fusion and late fusion to achieve the automatic weighting of the features and decisions.The hybrid fusion of early fusion and late fusion is implemented by using Dempster's rules of combination.The experimental results show that the proposed method displays an accuracy of 97.44% on the Chinese data set of Weibo and 92.35% on the data set of Twitter.The accuracy and F1-score of the method are both better than those of the base-line methods and advanced multimodal methods.

Key words: rumor detection, multi-modal fusion, attention mechanism, hybrid fusion, Dempster's combination rule

摘要： 社交媒体内容结构具有复杂性，大量虚假信息掺杂在真实内容中，或者在真实图片上配以杜撰的文字内容，导致基于单个模态的方法难以有效检测谣言。提出基于注意力机制与Dempster’s组合规则的混合融合方法。通过新增用户模态，提取文本、视觉和用户3个模态的特征向量，利用注意力机制对词语和视觉进行双向匹配，给予对谣言检测具有更多贡献的词语和视觉神经元更大的权值。在前后期融合均加入注意力机制，实现特征和决策的自动加权，并使用Dempster's组合规则实现混合融合。在真实的中文Weibo数据集和外文Twitter数据集上的实验结果表明，该方法准确率分别达到97.44%和92.35%，准确率和F1-score指标均高于基准方法和多模态方法。

关键词: 谣言检测, 多模态融合, 注意力机制, 混合融合, Dempster's组合规则

CLC Number:

TP181

TAO Xiao, ZHU Yan, LI Chunping. Rumor Detection Method Based on Attention and Multi-Modal Hybrid Fusion[J]. Computer Engineering, 2021, 47(12): 71-77.

陶霄, 朱焱, 李春平. 基于注意力与多模态混合融合的谣言检测方法[J]. 计算机工程, 2021, 47(12): 71-77.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0059683

http://www.ecice06.com/EN/Y2021/V47/I12/71

Figures/Tables 6

References

[1] CASTILLO C, MENDOZA M, POBLETE B.Information credibility on Twitter[C]//Proceedings of the 20th International Conference on World Wide Web.Hyderabad, India:[s.n.], 2011:675-684.
[2] MA J, GAO W, MITRA P, et al.Detecting rumors from microblogs with recurrent neural networks[C]//Proceedings of the 25th International Joint Conference on Artiﬁcial Intelligence.New York, USA:ACM Press, 2016:3818-3824.
[3] BHATT G, SHARMA A, SHARMA S, et al.Combining neural, statistical and external features for fake news identification[C]//Proceedings of International Conference of World Wide Web.Lyon, France:ACM Press, 2018:1353-1357.
[4] ZHIYUAN Z, YUAN Q, ZHENG L, et al.A C-GRU neural network for rumors detection[C]//Proceedings of the 5th International Conference on Cloud Computing and Intelligence Systems.Nanjing, China:[s.n.], 2018:704-708.
[5] QI P, CAO J, YANG T, et al.Exploiting multi-domain visual information for fake news detection[C]//Proceedings of 2019 International Conference on Data Mining.Beijing, China:[s.n.], 2019:518-527.
[6] BAHDANAU D, CHO K, BENGIO Y.Neural machine translation by jointly learning to align and translate[C]//Proceedings of 2015 International Conference on Learning Representations.San Diego, USA:ACM Press, 2015:1104-1116.
[7] VASWANI A, SHAZEER N.Attention is all you need[C]//Proceedings of Advances in Neural Information Processing Systems.Long Beach, USA:[s.n.], 2017:5998-6008.
[8] 何俊, 张彩庆, 李小珍, 等.面向深度学习的多模态融合技术研究综述[J].计算机工程, 2020, 46(5):1-11. HE J, ZHANG C Q, LI X Z, et al.Survey of research on multimodal fusion technology for deep learning[J].Computer Engineering, 2020, 46(5):1-11.(in Chinese)
[9] 冯耀功, 蔡国永.融合多层语义的跨模态检索[J].计算机科学, 2019, 46(3):227-233. FENG Y G, CAI G Y.Cross-modal retrieval fusing multilayer semantics[J].Computer Science, 2019, 46(3):227-233.(in Chinese)
[10] JIN Z W, CAO J, GUO H, et al.Multimodal fusion with recurrent neural networks for rumor detection on microblogs[C]//Proceedings of the 25th ACM International Conference on Multimedia.Mountain View, USA:ACM Press, 2017:795-816.
[11] DEVAMANYU H, SRUTHI G.Self-attentive feature-level fusion for multimodal emotion detection[C]//Proceedings of International Conference on Multimedia Information Processing and Retrieval.Miami, USA:ACM Press, 2018:196-201.
[12] 卢良锋, 谢志军, 叶宏武.基于RGB特征与深度特征融合的物体识别算法[J].计算机工程, 2016, 42(5):186-193. LU L F, XIE Z J, YE H W.Object recognition algorithm based on RGB feature and depth feature fusing[J].Computer Engineering, 2016, 42(5):186-193.(in Chinese)
[13] GENG Y, LIN Z, FU P, et al.Rumor detection on social media:a multi-view model using self-attention mechanism[C]//Proceedings of International Conference on Computational Science.Berlin, Germany:Springer, 2019:339-352.
[14] PENNINGTON J, SOCHER R, CHRISTOPHER D.GloVe:global vectors for word representation[C]//Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing.Qatar, Doha:Association for Computational Linguistics, 2014:1532-1543.
[15] DEVLIN J, CHANG M W.BERT:pre-training of deep bidirectional transformers for language understanding[EB/OL].[2020-09-01].https://arxiv.org/abs/1810.04805v1.
[16] MIKOLOV T, CHEN K.Efficient estimation of word representations in vector space[C]//Proceedings of International Conference on Learning Representations.New York, USA:ACM Press, 2013:1-8.
[17] DEMPSTER A P.A generalization of Bayesian inference[J].Journal of the Royal Statal Society, 1968, 30(2):205-232.
[18] WANG Y Q, MA F L.EANN:event adversarial neural networks for multi-modal fake news detection[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.London, UK:[s.n.], 2018:849-857.
[19] ALEX G, GREG W, IVO D.Neural Turing machines[EB/OL].[2020-09-01].https://arxiv.org/abs/1410.6247.
[20] THANG L, HIEU P, CHRISTOPHER D.Effective approaches to attention-based neural machine translation[C]//Proceedings of 2015 Conference on Empirical Methods in Natural Language Processing.Lisbon, Portugal:[s.n.], 2015:1412-1421.

Please choose a citation manager

Content to export