Review of Multimodal False Information Detection

doi:10.19678/j.issn.1000-3428.0253388

Abstract

Abstract:

In the digital era, complex interactions among modalities such as text, image, and audio have led to multimodal misinformation. The propagation speed and concealment levels of multimodal misinformation far exceed those of traditional unimodal misinformation, posing severe challenges to information security and social governance. However, research in this field is relatively scarce in China, and a comprehensive framework has yet to be established. Therefore, this paper provides a systematic overview of the research status and development trajectory of multimodal misinformation detection. Based on a clear understanding of the core concepts and task spectrum of multimodal misinformation detection, this review provides a detailed analysis of the characteristics of the datasets and evaluation metrics. It also analyzes the applicability and detection performance of different multimodal methods and models, such as SAFE, CAFE, CFFN, SSA-MFND, PSCC-Net, DGM4, CCN, SNIFFER, and KGAlign. Three core detection methods, namely cross-modal consistency, anomaly feature recognition, and external fact-driven approaches, are summarized. Furthermore, the interpretability and generalization robustness of multimodal misinformation detection are analyzed. With the rise of Large Vision-Language Models (LVLMs), their application to multimodal misinformation detection is continuously increasing. Various application scenarios, advantages, and limitations of LVLMs in this domain are discussed. Finally, future research directions in multimodal misinformation detection are outlined, aiming to provide insights and inspiration for further development in this field.

Key words: multimodality, false information detection, Large Vision-Language Model (LVLM), modal feature fusion, fact checking

摘要：

数字时代下, 文本、图像、音频等模态的复杂交互形成了多模态虚假信息, 其传播速度与隐蔽程度远超传统单模态虚假信息, 对信息安全与社会治理构成严峻挑战。但在国内该领域相关研究较为匮乏, 尚未形成完整体系。为此, 研究系统梳理了多模态虚假信息检测领域的研究现状及发展脉络, 对多模态虚假信息检测的研究进行了全面总结。在明确多模态虚假信息检测的核心概念与任务谱系的基础上, 详细总结了数据集与测评指标特征, 分析了SAFE、CAFE、CFFN、SSA-MFND、PSCC-Net、DGM4、CCN、SNIFFER、KGAlign等不同多模态方法模型的适用场景与检测性能, 归纳了跨模态一致性、异常特征识别、外部事实驱动三大核心检测方法, 并且对多模态虚假信息检测的可解释性与泛化鲁棒性进行了探讨。同时, 随着大规模视觉语言模型(LVLM)的崛起, 其在多模态虚假信息检测中的应用不断深化, 对此研究梳理了LVLM在该领域的多种应用场景、优势与局限。最后展望了多模态虚假信息检测的未来研究方向, 以期为多模态虚假信息检测领域的发展提供借鉴与启示。

关键词: 多模态, 虚假信息检测, 大规模视觉语言模型, 模态特征融合, 事实核查

HAO Guanyi, SUN Jingchao. Review of Multimodal False Information Detection[J]. Computer Engineering, 2026, 52(9): 64-80.

郝冠一, 孙靖超. 多模态虚假信息检测综述[J]. 计算机工程, 2026, 52(9): 64-80.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0253388

https://www.ecice06.com/EN/Y2026/V52/I9/64

Figures/Tables 10

Fig.1 Dimensions of multimodal misinformation

Fig.2 Evolution of multimodal misinformation detection task spectrum

Fig.3 RAG detection framework

Fig.4 Simulation of tool invocation process

Fig.5 Structure of multi-granularity attribution framework

Fig.6 Generalization and robustness issues in multimodal misinformation detection

Fig.7 Typical LVLM architecture and its application in multimodal misinformation detection

References 69

1	张欣, 孙靖超. 基于大语言模型的虚假信息检测框架综述. 计算机科学与探索, 2025, 19(6): 1414- 1436.
	ZHANG X , SUN J C . Review of false information detection frameworks based on large language models. Journal of Frontiers of Computer Science and Technology, 2025, 19(6): 1414- 1436.
2	VOSOUGHI S , ROY D , ARAL S . The spread of true and false news online. Science, 2018, 359(6380): 1146- 1151. doi: 10.1126/science.aap9559
3	CASTILLO C, MENDOZA M, POBLETE B. Information credibility on Twitter[C]//Proceedings of the 20th International Conference on World Wide Web. New York, USA: ACM Press, 2011: 675-684.
4	ZHOU P, HAN X, MORARIU V I, et al. Learning rich features for image manipulation detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2018: 1053-1061.
5	ROSSLER A, COZZOLINO D, VERDOLIVA L, et al. FaceForensics++: learning to detect manipulated facial images[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2019: 1-11.
6	WANG Y, MA F, JIN Z, et al. EANN: event adversarial neural networks for multi-modal fake news detection[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York, USA: ACM Press, 2018: 849-857.
7	ZHOU X Y, WU J D, ZAFARANI R. SAFE: similarity-aware multi-modal fake news detection[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2003.04981.
8	BRAUN T, ROTHERMEL M, ROHRBACH M, et al. DEFAME: dynamic evidence-based FAct-checking with multimodal experts[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2412.10510.
9	KAKIZAKI K, MATSUNAGA Y, FURUKAWA R. MAFT: multimodal automated fact-checking via textualization[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2025: 29646-29648.
10	许旻辰, 屈丹, 司念文, 等. 社交媒体虚假信息检测技术研究综述. 计算机工程, 2026, 52(5): 303- 325. doi: 10.19678/j.issn.1000-3428.0070287
	XU M C , QU D , SI N W , et al. A survey of research on social media disinformation detection technologies. Computer Engineering, 2026, 52(5): 303- 325. doi: 10.19678/j.issn.1000-3428.0070287
11	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the 38th International Conference on Machine Learning. [S.l.]: PMLR, 2021: 8748-8763.
12	LI J N, LI D X, SAVARESE S, et al. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2301.12597.
13	ZHANG S L, FANG Q K, YANG Z, et al. LLaVA-mini: efficient image and video large multimodal models with one vision token[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2501.03895.
14	DAI W L, LI J N, LI D X, et al. InstructBLIP: towards general-purpose vision-language models with instruction tuning[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2305.06500.
15	LIU H C, XUE W Y, CHEN Y F, et al. A survey on hallucination in large vision-language models[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2402.00253.
16	ALAM F, CRESCI S, CHAKRABORTY T, et al. A survey on multimodal disinformation detection[C]//Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju, Republic of Korea: International Committee on Computational Linguistics, 2022: 6625-6643.
17	LUO G, DARRELL T, ROHRBACH A. NewsCLIPpings: automatic generation of out-of-context multimodal media[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021: 6801-6817.
18	ANEJA S, BREGLER C, NIESSNER M. Catching out-of-context misinformation with self-supervised learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2021: 1342-1351.
19	TOLOSANA R , VERA-RODRIGUEZ R , FIERREZ J , et al. Deepfakes and beyond: a survey of face manipulation and fake detection. Information Fusion, 2020, 64, 131- 148. doi: 10.1016/j.inffus.2020.06.014
20	于凯, 宿天睿. 基于信息风险感知理论的虚假信息点对点传播建模与仿真研究. 计算机科学, 2023, 50(7): 376- 385.
	YU K , SU T R . Modeling and simulation of point-to-point propagation of false information based on information risk perception. Computer Science, 2023, 50(7): 376- 385.
21	POPAT K, MUKHERJEE S, STRÖTGEN J, et al. CredEye: a credibility lens for analyzing and explaining misinformation[C]//Proceedings of the Web Conference 2018. Geneva, Switzerland: International World Wide Web Conferences Steering Committee, 2018: 155-158.
22	SHU K, CUI L, WANG S, et al. dEFEND: explainable fake news detection[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York, USA: ACM Press, 2019: 395-405.
23	TAN R, PLUMMER B, SAENKO K. Detecting cross-modal inconsistency to defend against neural fake news[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Philadelphia, USA: ACL Press, 2020: 2081-2106.
24	WANG S Y, WANG O, ZHANG R, et al. CNN-generated images are surprisingly easy to spot… for now[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2020: 8692-8701.
25	GUO H, MA Z H, ZENG Z, et al. Each fake news is fake in its own way: an attribution multi-granularity benchmark for multimodal fake news detection[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2412.14686.
26	CUI X, APARCEDO A, JANG Y K, et al. On the robustness of large multimodal models against image adversarial attacks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2024: 24625-24634.
27	JIN Z, CAO J, GUO H, et al. Multimodal fusion with recurrent neural networks for rumor detection on microblogs[C]//Proceedings of the 25th ACM International Conference on Multimedia. New York, USA: ACM Press, 2017: 829-837.
28	NAKAMURA K, LEVY S, WANG W Y. Fakeddit: a new multimodal benchmark dataset for fine-grained fake news detection[C]//Proceedings of the 12th Language Resources and Evaluation Conference. Marseille, France: European Language Resources Association, 2020: 6149-6157.
29	SABIR E, ABDALMAGEED W, WU Y, et al. Deep multimodal image-repurposing detection[C]//Proceedings of the 26th ACM International Conference on Multimedia. New York, USA: ACM Press, 2018: 1337-1345.
30	SHAO R, WU T, LIU Z. Detecting and grounding multi-modal media manipulation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2023: 6904-6913.
31	QI P, BU Y Y, CAO J, et al. FakeSV: a multimodal benchmark with rich social context for fake news detection on short video platforms[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2023: 14444-14452.
32	BOIDIDOU C, ANDREADOU K, PAPADOPOULOS S, et al. Verifying multimedia use at mediaeval 2015[C]//Proceedings of MediaEval Benchmark 2015. Aachen, Germany: CEUR-WS, 2015: 1433-1436.
33	NAN Q, CAO J, ZHU Y C, et al. MDFEND: multi-domain fake news detection[C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management. New York, USA: ACM Press, 2021: 3343-3347.
34	CHEN Y X, LI D S, ZHANG P, et al. Cross-modal ambiguity learning for multimodal fake news detection[C]//Proceedings of the ACM Web Conference 2022. New York, USA: ACM Press, 2022: 2897-2905.
35	LI J, BIN Y, ZOU J, et al. Cross-modal consistency learning with fine-grained fusion network for multimodal fake news detection[C]//Proceedings of the ACM Multimedia Asia 2023. New York, USA: ACM Press, 2023: 1-7.
36	SHANG W Q , SONG K , JI J L , et al. Semantic space aligned multimodal fake news detection. Information Fusion, 2026, 125, 103469. doi: 10.1016/j.inffus.2025.103469
37	ZHANG L K , ZHANG C W , ZHANG Z W , et al. SAFE-GTA: semantic augmentation-based multimodal fake news detection via global-token attention. Symmetry, 2025, 17(6): 961. doi: 10.3390/sym17060961
38	LIU X H , LIU Y J , CHEN J , et al. PSCC-Net: progressive spatio-channel correlation network for image manipulation detection and localization. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 32(11): 7505- 7517.
39	KIM T, JEONG Y, CHOI J, et al. Beyond spatial frequency: pixel-wise temporal frequency-based deepfake video detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2024: 28221-28231.
40	ZHANG B L , YIN Q L , LU W , et al. Deepfake detection and localization using multi-view inconsistency measurement. IEEE Transactions on Dependable and Secure Computing, 2025, 22(2): 1796- 1809. doi: 10.1109/TDSC.2024.3472064
41	BAEVSKI A, ZHOU H, MOHAMED A, et al. wav2vec 2.0: a framework for self-supervised learning of speech representations[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2006.11477.
42	LEE S , CHOI S , KANG T , et al. iWAX: interpretable Wav2vec-AASIST-XGBoost framework for voice spoofing detection. Scientific Reports, 2025, 15, 40491. doi: 10.1038/s41598-025-24361-5
43	LI B H, LIANG Z Y, LIU J W, et al. Lips are lying: spotting the temporal inconsistency between audio and visual in lip-syncing DeepFakes[C]//Proceedings of the Advances in Neural Information Processing Systems 37. Vancouver, Canada: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2024: 91131-91155.
44	OORLOFF T, KOPPISETTI S, BONETTINI N, et al. AVFF: audio-visual feature fusion for video deepfake detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2024: 27092-27102.
45	ASTRID M, GHORBEL E, AOUADA D. Audio-visual deepfake detection with local temporal inconsistencies[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2501.08137.
46	ABDELNABI S, HASAN R, FRITZ M. Open-domain, content-based, multi-modal fact-checking of out-of-context images via online resources[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 14920-14929.
47	QI P, YAN Z H, HSU W, et al. SNIFFER: multimodal large language model for explainable out-of-context misinformation detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2024: 13052-13062.
48	LEWIS P, PEREZ E, PIKTUS A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2005.11401.
49	HU X M, GUO Z J, CHEN J Z, et al. MR2: a benchmark for multimodal retrieval-augmented rumor detection in social media[C]//Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, USA: ACM Press, 2023: 2901-2912.
50	CUI H R, REN M Y, ZHENG P, et al. A cross-domain knowledge graph-based cognitive inspiration and alignment method towards innovative design[C]//Proceedings of the IEEE 21st International Conference on Automation Science and Engineering (CASE). Washington D.C., USA: IEEE Press, 2025: 3361-3366.
51	DONG J, WANG W, TAN T N. CASIA image tampering detection evaluation database[C]//Proceedings of the IEEE China Summit and International Conference on Signal and Information Processing. Washington D.C., USA: IEEE Press, 2013: 422-426.
52	GUAN H Y, KOZAK M, ROBERTSON E, et al. MFC datasets: large-scale benchmark datasets for media forensic challenge evaluation[C]//Proceedings of the IEEE Winter Applications of Computer Vision Workshops (WACVW). Washington D.C., USA: IEEE Press, 2019: 63-72.
53	CUI X, ZOU Y Y, LI Z K, et al. T²Agent a tool-augmented multimodal misinformation detection agent with Monte Carlo tree search[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2505.19768.
54	MA J, GAO W, MITRA P, et al. Detecting rumors from microblogs with recurrent neural networks[C]//Proceedings of the International Joint Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2016: 3818-3824.
55	HARDALOV M, ARORA A, NAKOV P, et al. Few-shot cross-lingual stance detection with sentiment-based pre-training[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2022: 10729-10737.
56	WANG B, LI X, YE M, et al. Remember past, anticipate future: learning continual multimodal misinformation detectors[C]//Proceedings of the 33rd ACM International Conference on Multimedia. New York, USA: ACM Press, 2025: 5490-5498.
57	ZHANG W E , SHENG Q Z , ALHAZMI A , et al. Adversarial attacks on deep-learning models in natural language processing: a survey. ACM Transactions on Intelligent Systems and Technology, 2020, 11(3): 1- 41.
58	CARLINI N, WAGNER D. Towards evaluating the robustness of neural networks[C]// Proceedings of the IEEE Symposium on Security and Privacy (SP). Washington D.C., USA: IEEE Press, 2017: 39-57.
59	MUKHERJEE A, GHOSH S. UNITE-FND: reframing multimodal fake news detection through unimodal scene translation[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2502.11132.
60	ALAYRAC J B, BARR I, BARREIRA R, et al. Flamingo: a visual language model for few-shot learning[C]//Proceedings of the Advances in Neural Information Processing Systems 35. New Orleans, USA: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2022: 23716-23736.
61	LI Y F, DU Y F, ZHOU K, et al. Evaluating object hallucination in large vision-language models[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, 2023: 292-305.
62	WEIDINGER L, MELLOR J, RAUH M, et al. Ethical and social risks of harm from language models[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2112.04359.
63	MATERN F, RIESS C, STAMMINGER M. Exploiting visual artifacts to expose deepfakes and face manipulations[C]//Proceedings of the IEEE Winter Applications of Computer Vision Workshops (WACVW). Washington D.C., USA: IEEE Press, 2019: 83-92.
64	WANG Z D, BAO J M, ZHOU W G, et al. DIRE for diffusion-generated image detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2023: 22388-22398.
65	C2PA. C2PA specifications: V 1.0[EB/OL]. [2025-11-07]. https://spec.c2pa.org/specifications/specifications/1.0/index.html.
66	VATSA M, JAIN A, SINGH R. Adventures of trustworthy vision-language models: a survey[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2024: 22650-22658.
67	BIAN T, XIAO X, XU T Y, et al. Rumor detection on social media with bi-directional graph convolutional networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2020: 549-556.
68	HAN Y, KARUNASEKERA S, LECKIE C. Continual learning for fake news detection from social media[C]// Proceedings of ICANN 2021. Berlin, Germany: Springer, 2021: 372-384.
69	YAO S, ZHAO J, YU D, et al. ReAct: synergizing reasoning and acting in language models[EB/OL]. [2025-11-07]. https://arxiv.org/abs/2210.03629.

[1]	XU Minchen, QU Dan, SI Nianwen, PENG Sisi, CHEN Yaqi. Technologies for Detecting Disinformation in Social Media: A Comprehensive Review [J]. Computer Engineering, 2026, 52(5): 60-80.
[2]	CHEN Guolian, FENG Ziyang, CAO Junkuo. Research on Cyberbullying Detection Based on Multimodal Spatial Feature Fusion [J]. Computer Engineering, 2026, 52(3): 255-263.
[3]	Huayu LI, Zhikang ZHANG, Yang YAN, Yang YUE. Enhanced Domain Multi-modal Entity Recognition Based on Knowledge Graph [J]. Computer Engineering, 2024, 50(8): 31-39.
[4]	TANG Yuhao, MAO Qirong, GAO Lijian. Dimensional Emotion Recognition Method Based on Hierarchical Attention Mechanism [J]. Computer Engineering, 2020, 46(6): 65-72.
[5]	HE Jun, ZHANG Caiqing, LI Xiaozhen, ZHANG Dehai. Survey of Research on Multimodal Fusion Technology for Deep Learning [J]. Computer Engineering, 2020, 46(5): 1-11.

Please choose a citation manager

Content to export