计算机工程

• •    

基于注意力和多模态混合融合的谣言检测方法

  

  • 发布日期:2020-12-15

Rumor Detection with Attention and Multi-modal Hybrid Fusion

  • Published:2020-12-15

摘要: 社交媒体内容结构复杂,很多虚假信息掺杂在真实内容中或者真实图片配以杜撰的文字内容,基于单个模态难以有效检测谣言。为了有效解决这一问题,提出了基于注意力机制与Dempster组合规则的混合融合方法(DHF)。首先,新增用户模态,并提取文本、视觉和用户三个模态的特征向量。其次,利用注意力机制,对词语和视觉进行双向匹配,给予对谣言检测具有更多贡献的词语和视觉神经元更大的权值。最后,在前期融合和后期融合均加入注意力机制,实现特征和决策的自动加权,并使用Dempster组合规则实现混合融合。DHF在真实的中文Weibo数据集和外文Twitter数据集上分别达到了97.44%和92.35%的准确率,Accuracy和F1-score指标均超过了多个基准方法和两个先进的多模态方法。

Abstract: Owing to the complexity of content structure on social media, there are many rumors mixed in the real content. Similarly, there are many real pictures with fabricated content. So, it is difficult to detect rumors effectively based on single modal. In order to solve this problem, a hybrid fusion method based on attention mechanism and Dempster combination rules (DHF) was introduced. Firstly, the user modal was integrated, and the feature vectors of text modal, vision modal and user modal were extracted. Then, the attention mechanism was utilized to give more weight to words and visual neurons that contributed more to rumor detection. At last, the attention mechanism was added to early fusion and late fusion to achieve the automatic weighting of the features and decisions. The hybrid fusion of early fusion and late fusion was implemented by Dempster combination rules. The DHF achieves 97.44% and 92.35% accuracy on real Chinese Weibo data set and foreign Twitter data set respectively. The accuracy and F1 score are both better than the base-line methods and two advanced multimodal methods.