基于预训练和多模态融合的假新闻检测

doi:10.19678/j.issn.1000-3428.0066412

摘要/Abstract

摘要：

现有的多模态检测模型通常对每个模态的特征进行简单拼接，不能对模态之间的相关性进行有效建模，而且很难迁移到标签稀少的领域。提出一种基于预训练和多模态融合的假新闻检测模型PMFD。提取新闻附带图像不同区域的特征作为图像原始向量，合并图像原始向量作为图像引导向量，设计早期融合、中期融合、后期融合3种不同的多模态融合方式。在早期融合阶段，通过图像引导向量初始化文本特征提取器，获取文本原始向量，合并文本原始向量作为文本引导向量。在中期融合阶段，使用模态的原始向量集合与其他模态的引导向量构造模态的特征表示。在后期融合阶段，融合不同模态的特征表示，构造新闻的特征表示。为提高模型的泛化能力，在标签丰富的数据上对PMFD进行预训练，然后再在标签稀少的数据上对PMFD进行微调。在公开数据集上的实验结果表明，PMFD能有效检测假新闻结果，相对传统模型CNN、LSTM、BERT等有10%以上的提升，相对EANN、M_model多模态假新闻检测模型有2%~3%的提升。

关键词: 假新闻检测, 预训练, 多模态融合, 引导向量, 跨模态共享特征, 阶段融合

Abstract:

Existing multi-modal detection models are typically characterized by a simple splicing of features from each modality and are often ineffective in modeling the correlation between modalities. Furthermore, the migration of these models to domains with sparse labels is challenging. In this paper, a PMFD model, based on pre-training and multi-modal fusion, is proposed. Initially, image raw vectors are extracted from different regions of news incidental images, which are then merged to form image guide vectors. Three distinct multimodal fusion methods are designed: early fusion, middle fusion, and post fusion. During early fusion, the text feature extractor is initialized with image bootstrap vectors, leading to the acquisition of text original vectors, which are subsequently merged into text bootstrap vectors. In the middle fusion stage, the feature representation of the modality is constructed using the modality's original vectors combined with the bootstrap vectors of other modalities. For post fusion, the feature representations of different modalities are fused to construct the feature representation of news. To enhance the model's generalization capability, PMFD is initially pre-trained on label-rich data and then fine-tuned on label-sparse data. Experimental results on public data set show that, this approach demonstrates an improvement of over 10% compared to traditional models, including CNN, LSTM, and BERT, and a 2%-3% enhancement over existing EANN, M_model multi-modal fake news detection models.

Key words: fake news detection, pre-training, multi-modal fusion, bootstrap vector, cross-modal shared feature, stage fusion

周昊玮, 刘勇, 玄萍. 基于预训练和多模态融合的假新闻检测[J]. 计算机工程, 2024, 50(1): 289-295.

Haowei ZHOU, Yong LIU, Ping XUAN. Fake News Detection Based on Pre-Training and Multi-Modal Fusion[J]. Computer Engineering, 2024, 50(1): 289-295.

https://www.ecice06.com/CN/Y2024/V50/I1/289

图/表 9

图1 PMFD框架

Fig.1 Framework of PMFD

图2 词嵌入维度对模型性能的影响

Fig.2 Effect of word embedding dimension on model performance

图3 LSTM隐藏层大小对模型性能的影响

Fig.3 Effect of LSTM hidden layer size on model performance

参考文献 30

1	ZHOU X Y, ZAFARANI R, SHU K, et al. Fake news: fundamental theories, detection strategies and challenges[C]//Proceedings of the 12th ACM International Conference on Web Search and Data Mining. New York, USA: ACM Press, 2019: 836-837.
2	KWON S, CHA M, JUNG K, et al. Prominent features of rumor propagation in online social media[C]//Proceedings of 2013 IEEE International Conference on Data Mining. Washington D. C., USA: IEEE Press, 2013: 1103-1108.
3	ZHAO Z Y, ZHU H Y, XUE Z H, et al. An image-text consistency driven multimodal sentiment analysis approach for social media. Information Processing & Management, 2019, 56 (6): 102097. doi: 10.1016/j.ipm.2019.102097
4	段大高, 白宸宇, 韩忠明, 等. 基于多传递影响力的社交媒体谣言检测方法. 计算机工程, 2022, 48 (10): 138-145, 157. URL
	DUAN D G, BAI C Y, HAN Z M, et al. Social media rumor detection method based on multi-transmission influence. Computer Engineering, 2022, 48 (10): 138-145, 157. URL
5	QI P, CAO J A, YANG T Y, et al. Exploiting multi-domain visual information for fake news detection[C]//Proceedings of 2019 IEEE International Conference on Data Mining. Washington D. C., USA: IEEE Press, 2019: 518-527.
6	陶霄, 朱焱, 李春平. 基于注意力与多模态混合融合的谣言检测方法. 计算机工程, 2021, 47 (12): 71- 77. URL
	TAO X, ZHU Y, LI C P. Rumor detection method based on attention and multi-modal hybrid fusion. Computer Engineering, 2021, 47 (12): 71- 77. URL
7	RUCHANSKY N, SEO S, LIU Y. CSI: a hybrid deep model for fake news detection[C]//Proceedings of 2017 ACM Conference on Information and Knowledge Management. New York, USA: ACM Press, 2017: 797-806.
8	ZADEH A, CHEN M H, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[EB/OL]. [2022-10-10]. https://arxiv.org/abs/1707.07250.pdf.
9	GU Y, YANG K N, FU S Y, et al. Multimodal affective analysis using hierarchical attention strategy with word-level alignment[C]//Proceedings of Annual Meeting of the Association for Computational Linguistics. New York, USA: ACM Press, 2018: 2225-2235.
10	MA J, GAO W, WONG K F. Detect rumors on Twitter by promoting information campaigns with generative adversarial learning[C]//Proceedings of the World Wide Web Conference. New York, USA: ACM Press, 2019: 3049-3055.
11	ZHOU K, SHU C, LI B, et al. Early rumour detection[C]//Proceedings of 2019 Conference of the North American Chapter of the Association for Computational Linguistics. Stroudsburg, USA: Association for Computational Linguistics, 2019: 1614-1623.
12	CHEN Y X, SUI J, HU L, et al. Attention-residual network with CNN for rumor detection[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management. New York, USA: ACM Press, 2019: 1121-1130.
13	ERHAN D, COURVILLE A, BENGIO Y, et al. Why does unsupervised pre-training help deep learning?[C]//Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. [S. l.]: JMLR, 2010: 201-208.
14	GANIN Y, LEMPITSKY V. Unsupervised domain adaptation by backpropagation[EB/OL]. [2022-10-10]. https://arxiv.org/abs/1409.7495.pdf.
15	YANG F, LIU Y, YU X H, et al. Automatic detection of rumor on Sina Weibo[C]//Proceedings of ACM SIGKDD Workshop on Mining Data Semantics. New York, USA: ACM Press, 2012: 1-7.
16	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks. Communications of the ACM, 2020, 63 (11): 139- 144. doi: 10.1145/3422622
17	GU Y, YANG K N, FU S Y, et al. Hybrid attention based multimodal network for spoken language classification[C]//Proceedings of Annual Meeting of the Association for Computational Linguistics. New York, USA: ACM Press, 2018: 2379-2390.
18	PENNINGTON J, SOCHER R, MANNING C. GloVe: global vectors for word representation[C]//Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: Association for Computational Linguistics, 2014: 1532-1543.
19	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2022-10-10]. https://arxiv.org/abs/1810.04805.pdf.
20	FLORIDI L, CHIRIATTI M. GPT-3: its nature, scope, limits, and consequences. Minds and Machines, 2020, 30 (4): 681- 694. doi: 10.1007/s11023-020-09548-1
21	PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C. USA: IEEE Press, 2016: 2536-2544.
22	LI J L, XU Y H, LV T C, et al. DiT: self-supervised pre-training for document image transformer[EB/OL]. [2022-10-10]. https://arxiv.org/abs/1810.04805.pdf.
23	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 770-778.
24	ZHOU P, SHI W, TIAN J, et al. Attention-based bidirectional long short-term memory networks for relation classification[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, USA: Association for Computational Linguistics, 2016: 207-212.
25	SONG C G, NING N W, ZHANG Y L, et al. A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks. Information Processing & Management, 2021, 58 (1): 102437. doi: 10.1016/j.ipm.2020.102437
26	LECUN Y, BOSER B, DENKER J S, et al. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1989, 1 (4): 541- 551. doi: 10.1162/neco.1989.1.4.541
27	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 6000-6010.
28	SUN Z J, LI X Y, SUN X F, et al. ChineseBERT: Chinese pretraining enhanced by glyph and pinyin information[EB/OL]. [2022-10-10]. https://arxiv.org/abs/2106.16038.pdf.
29	WANG Y Q, MA F L, JIN Z W, et al. EANN: event adversarial neural networks for multi-modal fake news detection[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York, USA: ACM, Press, 2018: 849-857.
30	CAI Y T, CAI H Y, WAN X J. Multi-modal sarcasm detection in Twitter with hierarchical fusion model[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, USA: Association for Computational Linguistics, 2019: 2506-2515.

[1]	魏嵬, 丁香香, 郭梦星, 杨钊, 刘辉. 文本相似度计算方法综述[J]. 计算机工程, 2024, 50(9): 18-32.
[2]	陈宇航, 杨勇, 先木斯亚·买买提明, 帕力旦·吐尔逊, 樊小超, 任鸽, 刁宇峰. 基于主题感知和语义增强的作文自动评分方法[J]. 计算机工程, 2024, 50(8): 363-371.
[3]	刘娟, 段友祥, 陆誉翕, 张鲁. 引入知识增强和对比学习的知识图谱补全[J]. 计算机工程, 2024, 50(7): 112-122.
[4]	武星, 殷浩宇, 姚骏峰, 李卫民, 钱权. 面向视频数据的多模态情感分析[J]. 计算机工程, 2024, 50(6): 218-227.
[5]	陈佳玉, 王元龙, 张虎. 基于文本知识增强的问题生成模型[J]. 计算机工程, 2024, 50(6): 86-93.
[6]	隗昊, 刁宏悦, 孔亮宸, 邓耀臣. 东北亚舆情文本细粒度命名实体识别方法研究[J]. 计算机工程, 2024, 50(5): 354-362.
[7]	隗昊, 刁宏悦, 孔亮宸, 邓耀臣. 东北亚舆情文本细粒度命名实体识别方法研究[J]. 计算机工程, 2024, 50(5): 354-362.
[8]	李田芳, 普园媛, 赵征鹏, 徐丹, 钱文华. 基于CLIP和双空间自适应归一化的图像翻译[J]. 计算机工程, 2024, 50(5): 229-240.
[9]	周昭辰, 方清茂, 吴晓红, 胡平, 何小海. 基于MacBERT与对抗训练的机器阅读理解模型[J]. 计算机工程, 2024, 50(5): 41-50.
[10]	侯钰涛, 阿布都克力木·阿布力孜, 史亚庆, 马依拉木·木斯得克, 哈里旦木·阿布都克里木. 面向"一带一路"的低资源语言机器翻译研究[J]. 计算机工程, 2024, 50(4): 332-341.
[11]	王琳, 黄浩. 引入预训练表示混合矢量量化和CTC的语音转换[J]. 计算机工程, 2024, 50(4): 313-320.
[12]	张洪程, 李林育, 杨莉, 伞晨峻, 尹春林, 颜冰, 于虹, 张璇. 基于对比学习与语言模型增强嵌入的知识图谱补全[J]. 计算机工程, 2024, 50(4): 168-176.
[13]	于明诚, 党亚固, 吴奇林, 吉旭, 毕可鑫. 基于多尺度上下文的英文作文自动评分研究[J]. 计算机工程, 2024, 50(3): 259-266.
[14]	朱贵德, 黄海. 文本视觉问答综述[J]. 计算机工程, 2024, 50(2): 1-14.
[15]	施竣潇, 陈艳平, 穆肇南. 融合多尺度跨度特征的谓语中心词识别模型[J]. 计算机工程, 2024, 50(10): 137-144.

选择文件类型/文献管理软件名称

选择包含的内容