Dual-Stream Generalized Face Forgery Detection Method Based on Intra-Inter Frame Self-Blending

doi:10.19678/j.issn.1000-3428.0068356

Abstract

Abstract:

Although existing methods for detecting face forgery perform well within familiar source domains, they often suffer from overfitting, leading to a lack of generalizability in face forgery detection. As a result, their performance significantly decreases when faced with unfamiliar or unknown scenarios. To address this issue, this study proposes a dual-stream generalized face forgery detection method based on intra-inter frame self-blending. The intra-inter frame self-blending module is designed to prevent detector overfitting from unrelated identity information by leveraging inconsistencies within and between frames to generate a diverse and realistic forgery training set. This method enhances the generalizability of the detection model. Additionally, a detection model is developed as a dual-stream network using an RGB-frequency feature-enhancing module, which extracts and fuses RGB and high-frequency features within the shallow layers of the network to capture forged artifacts. This method not only enhances the model performance but also alleviates the increase in model parameter size. Experiments are conducted against nine mainstream methods across four datasets, with the proposed model improving the AUC by 1.52% and EER by 1.5% on average in cross-dataset experiments. In addition, it ranks first or second in all four sub-datasets of different manipulations in cross-manipulation experiments. These results demonstrate that the proposed method achieves excellent generalizability in face forgery detection.

Key words: face forgery detection, inter-intra frame self-blending, feature fusion, attention mechanism, dual-stream network, generalization ability

摘要：

现有人脸伪造检测方法往往在已知伪造类型上表现良好, 但面对未知数据时检测性能有所下降, 模型易受到过拟合的影响, 检测泛化性不足。针对此问题, 提出一种基于帧内-帧间自融合的双流泛化人脸伪造检测方法, 从数据增强和检测器改进2个方面提高检测泛化性。设计帧内-帧间自融合模块, 分别利用同帧人脸、帧间人脸进行数据增强: 帧内自融合子模块利用同帧人脸生成训练数据, 从而避免人脸图像身份信息干扰; 帧间自融合子模块利用伪造视频的帧间不一致性, 进一步构造多样性丰富、逼真的训练数据集, 从而有效防止模型的过拟合, 确保检测模型的泛化能力。此外, 设计基于通道注意力机制的双流特征融合网络, 在网络的浅层提取RGB特征、高频特征并进行融合来挖掘伪造信息, 在提升模型性能的同时缓解网络的参数增长。将模型在4个数据集上与9种主流检测方法进行对比实验, 结果表明: 在跨数据集实验中, 所提方法较次优方法AUC均值提高1.52个百分点, EER均值降低1.5个百分点; 在跨伪造方法实验中, 所提方法在4种伪造方法子数据集上均取得最优或次优效果。实验结果验证了该方法优秀的泛化能力。

关键词: 人脸伪造检测, 帧内-帧间自融合, 特征融合, 注意力机制, 双流网络, 泛化能力

DONG Fengkai, ZOU Xiaoqiang, WANG Jiahui, MA Liming, YANG Wenyuan, LIU Xiyao. Dual-Stream Generalized Face Forgery Detection Method Based on Intra-Inter Frame Self-Blending[J]. Computer Engineering, 2024, 50(10): 185-195.

董丰恺, 邹晓强, 王佳慧, 马利民, 杨文元, 刘熙尧. 基于帧内-帧间自融合的双流泛化人脸伪造检测方法[J]. 计算机工程, 2024, 50(10): 185-195.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0068356

https://www.ecice06.com/EN/Y2024/V50/I10/185

Figures/Tables 13

Fig.1 Procedure of dual-stream generalized face forgery detection method based on intra-inter frame self-blending

Fig.2 Examples of intra frame self-blending images

Fig.3 Examples of inter frame self-blending images

Fig.4 Framework of the dual-stream feature fused detection network based on channel attention mechanism

Fig.5 Grad-CAM heatmaps comparison of I2FS and SBIs methods from different types of forgery of FF++

References 39

1	NIRKIN Y, MASI I, TUAN A T, et al. On face segmentation, face swapping, and face perception[C]//Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition. Washington D. C., USA: IEEE Press, 2018: 98-105.
2	JUEFEI-XU F, WANG R, HUANG Y H, et al. Countering malicious DeepFakes: survey, battleground, and horizon. International Journal of Computer Vision, 2022, 130 (7): 1678- 1734. doi: 10.1007/s11263-022-01606-8
3	孙毅, 王志浩, 邓佳, 等. 人脸深度伪造检测综述. 信息安全研究, 2022, 8 (3): 241- 257. doi: 10.12379/j.issn.2096-1057.2022.03.05
	SUN Y, WANG Z H, DENG J, et al. A survey of deep face forgery detection. Journal of Information Security Research, 2022, 8 (3): 241- 257. doi: 10.12379/j.issn.2096-1057.2022.03.05
4	谢天, 于灵云, 罗常伟, 等. 深度人脸伪造与检测技术综述. 清华大学学报(自然科学版), 2023, 63 (9): 1350- 1365. URL
	XIE T, YU L Y, LUO C W, et al. Survey of deep face manipulation and fake detection. Journal of Tsinghua University(Science and Technology), 2023, 63 (9): 1350- 1365. URL
5	LI M, LIU B B, HU Y J, et al. Exposing DeepFake videos by tracking eye movements[C]//Proceedings of the 25th International Conference on Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 5184-5189.
6	LI Y, LYU S. Exposing DeepFake videos by detecting face warping artifacts[EB/OL]. [2023-05-10]. https://arxiv.org/abs/1811.00656.
7	DANG H, LIU F, STEHOUWER J, et al. On the detection of digital face manipulation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 5781-5790.
8	NGUYEN H H, FANG F M, YAMAGISHI J, et al. Multi-task learning for detecting and segmenting manipulated facial images and videos[C]//Proceedings of the IEEE International Conference on Biometrics Theory, Applications and Systems. Washington D. C., USA: IEEE Press, 2019: 1-8.
9	LI L Z, BAO J M, ZHANG T, et al. Face X-ray for more general face forgery detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 5001-5010.
10	WANG J K, WU Z X, OUYANG W H, et al. M2TR: multi-modal multi-scale transformers for Deepfake detection[C]//Proceedings of the 2022 International Conference on Multimedia Retrieval. New York, USA: ACM Press, 2022: 615-623.
11	QIAN Y Y, YIN G J, SHENG L, et al. Thinking in frequency: face forgery detection by mining frequency-aware clues[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 86-103.
12	XU Y, TERHÖRST P, RAJA K, et al. A comprehensive analysis of AI biases in DeepFake detection with massively annotated databases[EB/OL]. [2023-05-10]. https://arxiv.org/abs/2208.05845v1.
13	SHIOHARA K, YAMASAKI T. Detecting deepfakes with self-blended images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 18699-18708.
14	LI D, YANG Y X, SONG Y Z, et al. Learning to generalize: meta-learning for domain generalization. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32 (1): 1- 10. doi: 10.48550/arXiv.1710.03463
15	SUN K, YAO T P, CHEN S, et al. Dual contrastive learning for general face forgery detection. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36 (2): 2316- 2324. doi: 10.1609/aaai.v36i2.20130
16	MALIK A, KURIBAYASHI M, ABDULLAHI S M, et al. DeepFake detection for human face images and videos: a survey. IEEE Access, 2022, 10, 18757- 18775. doi: 10.1109/ACCESS.2022.3151186
17	PEROV I, GAO D, CHERVONIY N, et al. DeepFaceLab: integrated, flexible and extensible face-swapping framework[EB/OL]. [2023-05-10]. https://arxiv.org/pdf/2005.05535v5.
18	JUNG T, KIM S, KIM K. DeepVision: Deepfakes detection using human eye blinking pattern. IEEE Access, 2020, 8, 83144- 83154. doi: 10.1109/ACCESS.2020.2988660
19	ELHASSAN A, AL-FAWA'REH M, JAFAR M T, et al. DFT-MF: enhanced deepfake detection using mouth movement and transfer learning. SoftwareX, 2022, 19, 101115. doi: 10.1016/j.softx.2022.101115
20	ROSSLER A, COZZOLINO D, VERDOLIVA L, et al. FaceForensics++: learning to detect manipulated facial images[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 1-11.
21	ZHAO H, WEI T, ZHOU W, et al. Multi-attentional deepfake detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 2185-2194.
22	ZHU X Y, WANG H, FEI H Y, et al. Face forgery detection by 3D decomposition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 2928-2938.
23	耿鹏志, 樊红兴, 张翌阳, 等. 基于篡改伪影的深度伪造检测方法. 计算机工程, 2021, 47 (12): 156- 162. doi: 10.19678/j.issn.1000-3428.0060733
	GENG P Z, FAN H X, ZHANG Y Y, et al. Deepfake detection method based on tampering artifacts. Computer Engineering, 2021, 47 (12): 156- 162. doi: 10.19678/j.issn.1000-3428.0060733
24	ZHOU P, HAN X T, MORARIU V I, et al. Learning rich features for image manipulation detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 1053-1061.
25	FRIDRICH J, KODOVSKY J. Rich models for steganalysis of digital images. IEEE Transactions on Information Forensics and Security, 2012, 7 (3): 868- 882. doi: 10.1109/TIFS.2012.2190402
26	LUO Y C, ZHANG Y, YAN J C, et al. Generalizing face forgery detection with high-frequency features[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 16312-16321.
27	SUN K, LIU H, YE Q X, et al. Domain general face forgery detection by learning to weight. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35 (3): 2638- 2646. doi: 10.1609/aaai.v35i3.16367
28	马欣, 吉立新, 李邵梅. 基于多尺度Transformer融合多域信息的伪造人脸检测. 计算机科学, 2023, 50 (10): 112- 118. doi: 10.11896/jsjkx.220900048
	MA X, JI L X, LI S M. Forgery face detection based on multi-scale transformer fusing multi-domain information. Computer Science, 2023, 50 (10): 112- 118. doi: 10.11896/jsjkx.220900048
29	KING D E. Dlib-ml: a machine learning toolkit. Journal of Machine Learning Research, 2009, 10, 1755- 1758. doi: 10.1145/1577069.1755843
30	LI Y Z, YANG X, SUN P, et al. Celeb-DF: a new dataset for DeepFake forensics[EB/OL]. [2023-05-10]. https://arxiv.org/abs/1909.12962.
31	DOLHANSKY B, BITTON J, PFLAUM B, et al. The DeepFake Detection Challenge(DFDC) dataset[EB/OL]. [2023-05-10]. https://arxiv.org/pdf/2006.07397.
32	Google. Deepfake detecttion dataset[EB/OL]. (2019-09-24)[2023-05-10]. https://ai.googleblog.com/2019/09/contributing-data-to-deepfake-detection.html.
33	DENG J K, GUO J, VERVERAS E, et al. RetinaFace: single-shot multi-level face localisation in the wild[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 5203-5212.
34	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60 (6): 84- 90. doi: 10.1145/3065386
35	TAN M, LE Q. EfficientNet: rethinking model scaling for convolutional neural networks[C]//Proceedings of International Conference on Machine Learning. [S. l.]: PMLR, 2019: 6105-6114.
36	CHEN S, YAO T P, CHEN Y, et al. Local relation learning for face forgery detection. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35 (2): 1081- 1088. doi: 10.1609/aaai.v35i2.16193
37	YADAV D, SALMANI S. Deepfake: a survey on facial forgery technique using generative adversarial network[C]//Proceedings of the International Conference on Intelligent Computing and Control Systems. Washington D. C., USA: IEEE Press, 2019: 852-857.
38	TOLOSANA R, VERA-RODRIGUEZ R, FIERREZ J, et al. Deepfakes and beyond: a survey of face manipulation and fake detection. Information Fusion, 2020, 64, 131- 148. doi: 10.1016/j.inffus.2020.06.014
39	SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 2020, 128 (2): 336- 359. doi: 10.1007/s11263-019-01228-7

[1]	LI Junjun, DONG Jiangang, LI Kun. Research on Kubernetes-based Cluster Energy-Saving Strategy [J]. Computer Engineering, 2024, 50(9): 82-91.
[2]	LIN Chang, GUO Wei, REN Zhecong, JIN Haibo. Unification Algorithm for Object Tracking and Segmentation Based on Transformer [J]. Computer Engineering, 2024, 50(9): 130-141.
[3]	LI Zelin, LÜ Zhaofeng, CHEN Fuqiang, LI Ke. Entity Alignment Model Based on Multi-Hop Information Fusion [J]. Computer Engineering, 2024, 50(9): 142-152.
[4]	WANG Ruying, MA Jiajun, DONG Jianqiang, LIU Wanlong, ZHANG Haitao, YIN Kai, ZHAO Bochao. Industrial Load Forecasting Method Based on MTS-BiGRU-DMHSA [J]. Computer Engineering, 2024, 50(9): 169-178.
[5]	LI Junyi, LI Xiangyang, LONG Chaoxun, LI Haiyan, LI Hongsong, YU Pengfei. Wild Mushroom Classification Based on Multi-level Region Selection and Cross-layer Feature Fusion [J]. Computer Engineering, 2024, 50(9): 179-188.
[6]	ZHU Kai, LI Li, ZHANG Tong, JIANG Sheng, BIE Yiming. Multi-Stage Motion Blur Image Restoration Network Based on Transformer [J]. Computer Engineering, 2024, 50(9): 276-285.
[7]	ZHANG Tianpeng, HAN Jing, LÜ Xueqiang. Super-Resolution-Aided Small-Target Detection Based on Multi-Task Learning [J]. Computer Engineering, 2024, 50(9): 304-312.
[8]	GUO Min, ZHANG Xihan, LI Yang. Integrated Attentional Teacher Mutual Consistency Semi-Supervised Medical Image Segmentation [J]. Computer Engineering, 2024, 50(9): 313-323.
[9]	ZENG Yuqi, LIU Bo, ZHONG Baichang, ZHONG Jin. Student Classroom Behavior Detection Algorithm Based on Improved YOLOv8 in Smart Education [J]. Computer Engineering, 2024, 50(9): 344-355.
[10]	Huaqing ZHANG, Zhangtao XIA, Xiaoqing LU, Jijun TONG. Named Entity Recognition of Vascular Surgery Based on Glyph Features [J]. Computer Engineering, 2024, 50(8): 13-21.
[11]	Rixin RAO, Yiwen WANG, Lizhi ZENG, Xintian TONG, Haitao ZHAO. Lightweight Network Model for Waste Cable Detection [J]. Computer Engineering, 2024, 50(8): 22-30.
[12]	Huayu LI, Zhikang ZHANG, Yang YAN, Yang YUE. Enhanced Domain Multi-modal Entity Recognition Based on Knowledge Graph [J]. Computer Engineering, 2024, 50(8): 31-39.
[13]	Lei WANG, Shipeng DANG, Feng PAN. Model for Predicting Concealed Accessory Pathway Based on Convolutional Neural Network [J]. Computer Engineering, 2024, 50(8): 40-49.
[14]	Han CHEN, Chunlei ZHAO, Haoda JIANG, Chundong WANG. Research on App User Intent Recognition Based on Fusion Model and Semantic Network [J]. Computer Engineering, 2024, 50(8): 50-63.
[15]	Suolan LIU, Yan WANG, Hongyuan WANG, Shengsheng ZHU. Human Behavior Recognition Based on Multi-Stream Semantic Graph Convolutional Network [J]. Computer Engineering, 2024, 50(8): 64-74.

Please choose a citation manager

Content to export