基于帧内-帧间自融合的双流泛化人脸伪造检测方法

doi:10.19678/j.issn.1000-3428.0068356

摘要/Abstract

摘要：

现有人脸伪造检测方法往往在已知伪造类型上表现良好, 但面对未知数据时检测性能有所下降, 模型易受到过拟合的影响, 检测泛化性不足。针对此问题, 提出一种基于帧内-帧间自融合的双流泛化人脸伪造检测方法, 从数据增强和检测器改进2个方面提高检测泛化性。设计帧内-帧间自融合模块, 分别利用同帧人脸、帧间人脸进行数据增强: 帧内自融合子模块利用同帧人脸生成训练数据, 从而避免人脸图像身份信息干扰; 帧间自融合子模块利用伪造视频的帧间不一致性, 进一步构造多样性丰富、逼真的训练数据集, 从而有效防止模型的过拟合, 确保检测模型的泛化能力。此外, 设计基于通道注意力机制的双流特征融合网络, 在网络的浅层提取RGB特征、高频特征并进行融合来挖掘伪造信息, 在提升模型性能的同时缓解网络的参数增长。将模型在4个数据集上与9种主流检测方法进行对比实验, 结果表明: 在跨数据集实验中, 所提方法较次优方法AUC均值提高1.52个百分点, EER均值降低1.5个百分点; 在跨伪造方法实验中, 所提方法在4种伪造方法子数据集上均取得最优或次优效果。实验结果验证了该方法优秀的泛化能力。

关键词: 人脸伪造检测, 帧内-帧间自融合, 特征融合, 注意力机制, 双流网络, 泛化能力

Abstract:

Although existing methods for detecting face forgery perform well within familiar source domains, they often suffer from overfitting, leading to a lack of generalizability in face forgery detection. As a result, their performance significantly decreases when faced with unfamiliar or unknown scenarios. To address this issue, this study proposes a dual-stream generalized face forgery detection method based on intra-inter frame self-blending. The intra-inter frame self-blending module is designed to prevent detector overfitting from unrelated identity information by leveraging inconsistencies within and between frames to generate a diverse and realistic forgery training set. This method enhances the generalizability of the detection model. Additionally, a detection model is developed as a dual-stream network using an RGB-frequency feature-enhancing module, which extracts and fuses RGB and high-frequency features within the shallow layers of the network to capture forged artifacts. This method not only enhances the model performance but also alleviates the increase in model parameter size. Experiments are conducted against nine mainstream methods across four datasets, with the proposed model improving the AUC by 1.52% and EER by 1.5% on average in cross-dataset experiments. In addition, it ranks first or second in all four sub-datasets of different manipulations in cross-manipulation experiments. These results demonstrate that the proposed method achieves excellent generalizability in face forgery detection.

Key words: face forgery detection, inter-intra frame self-blending, feature fusion, attention mechanism, dual-stream network, generalization ability

董丰恺, 邹晓强, 王佳慧, 马利民, 杨文元, 刘熙尧. 基于帧内-帧间自融合的双流泛化人脸伪造检测方法[J]. 计算机工程, 2024, 50(10): 185-195.

DONG Fengkai, ZOU Xiaoqiang, WANG Jiahui, MA Liming, YANG Wenyuan, LIU Xiyao. Dual-Stream Generalized Face Forgery Detection Method Based on Intra-Inter Frame Self-Blending[J]. Computer Engineering, 2024, 50(10): 185-195.

https://www.ecice06.com/CN/Y2024/V50/I10/185

图/表 13

图1 基于帧内-帧间自融合的双流泛化人脸伪造检测方法流程

Fig.1 Procedure of dual-stream generalized face forgery detection method based on intra-inter frame self-blending

图2 帧内自融合图片示例

Fig.2 Examples of intra frame self-blending images

图3 帧间自融合图片示例

Fig.3 Examples of inter frame self-blending images

图4 基于通道注意力机制的双流特征融合检测网络框架

Fig.4 Framework of the dual-stream feature fused detection network based on channel attention mechanism

图5 I2FS方法和SBIs方法在FF++4种伪造类型上的Grad-CAM热力图对比

Fig.5 Grad-CAM heatmaps comparison of I2FS and SBIs methods from different types of forgery of FF++

参考文献 39

1	NIRKIN Y, MASI I, TUAN A T, et al. On face segmentation, face swapping, and face perception[C]//Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition. Washington D. C., USA: IEEE Press, 2018: 98-105.
2	JUEFEI-XU F, WANG R, HUANG Y H, et al. Countering malicious DeepFakes: survey, battleground, and horizon. International Journal of Computer Vision, 2022, 130 (7): 1678- 1734. doi: 10.1007/s11263-022-01606-8
3	孙毅, 王志浩, 邓佳, 等. 人脸深度伪造检测综述. 信息安全研究, 2022, 8 (3): 241- 257. doi: 10.12379/j.issn.2096-1057.2022.03.05
	SUN Y, WANG Z H, DENG J, et al. A survey of deep face forgery detection. Journal of Information Security Research, 2022, 8 (3): 241- 257. doi: 10.12379/j.issn.2096-1057.2022.03.05
4	谢天, 于灵云, 罗常伟, 等. 深度人脸伪造与检测技术综述. 清华大学学报(自然科学版), 2023, 63 (9): 1350- 1365. URL
	XIE T, YU L Y, LUO C W, et al. Survey of deep face manipulation and fake detection. Journal of Tsinghua University(Science and Technology), 2023, 63 (9): 1350- 1365. URL
5	LI M, LIU B B, HU Y J, et al. Exposing DeepFake videos by tracking eye movements[C]//Proceedings of the 25th International Conference on Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 5184-5189.
6	LI Y, LYU S. Exposing DeepFake videos by detecting face warping artifacts[EB/OL]. [2023-05-10]. https://arxiv.org/abs/1811.00656.
7	DANG H, LIU F, STEHOUWER J, et al. On the detection of digital face manipulation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 5781-5790.
8	NGUYEN H H, FANG F M, YAMAGISHI J, et al. Multi-task learning for detecting and segmenting manipulated facial images and videos[C]//Proceedings of the IEEE International Conference on Biometrics Theory, Applications and Systems. Washington D. C., USA: IEEE Press, 2019: 1-8.
9	LI L Z, BAO J M, ZHANG T, et al. Face X-ray for more general face forgery detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 5001-5010.
10	WANG J K, WU Z X, OUYANG W H, et al. M2TR: multi-modal multi-scale transformers for Deepfake detection[C]//Proceedings of the 2022 International Conference on Multimedia Retrieval. New York, USA: ACM Press, 2022: 615-623.
11	QIAN Y Y, YIN G J, SHENG L, et al. Thinking in frequency: face forgery detection by mining frequency-aware clues[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 86-103.
12	XU Y, TERHÖRST P, RAJA K, et al. A comprehensive analysis of AI biases in DeepFake detection with massively annotated databases[EB/OL]. [2023-05-10]. https://arxiv.org/abs/2208.05845v1.
13	SHIOHARA K, YAMASAKI T. Detecting deepfakes with self-blended images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 18699-18708.
14	LI D, YANG Y X, SONG Y Z, et al. Learning to generalize: meta-learning for domain generalization. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32 (1): 1- 10. doi: 10.48550/arXiv.1710.03463
15	SUN K, YAO T P, CHEN S, et al. Dual contrastive learning for general face forgery detection. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36 (2): 2316- 2324. doi: 10.1609/aaai.v36i2.20130
16	MALIK A, KURIBAYASHI M, ABDULLAHI S M, et al. DeepFake detection for human face images and videos: a survey. IEEE Access, 2022, 10, 18757- 18775. doi: 10.1109/ACCESS.2022.3151186
17	PEROV I, GAO D, CHERVONIY N, et al. DeepFaceLab: integrated, flexible and extensible face-swapping framework[EB/OL]. [2023-05-10]. https://arxiv.org/pdf/2005.05535v5.
18	JUNG T, KIM S, KIM K. DeepVision: Deepfakes detection using human eye blinking pattern. IEEE Access, 2020, 8, 83144- 83154. doi: 10.1109/ACCESS.2020.2988660
19	ELHASSAN A, AL-FAWA'REH M, JAFAR M T, et al. DFT-MF: enhanced deepfake detection using mouth movement and transfer learning. SoftwareX, 2022, 19, 101115. doi: 10.1016/j.softx.2022.101115
20	ROSSLER A, COZZOLINO D, VERDOLIVA L, et al. FaceForensics++: learning to detect manipulated facial images[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 1-11.
21	ZHAO H, WEI T, ZHOU W, et al. Multi-attentional deepfake detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 2185-2194.
22	ZHU X Y, WANG H, FEI H Y, et al. Face forgery detection by 3D decomposition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 2928-2938.
23	耿鹏志, 樊红兴, 张翌阳, 等. 基于篡改伪影的深度伪造检测方法. 计算机工程, 2021, 47 (12): 156- 162. doi: 10.19678/j.issn.1000-3428.0060733
	GENG P Z, FAN H X, ZHANG Y Y, et al. Deepfake detection method based on tampering artifacts. Computer Engineering, 2021, 47 (12): 156- 162. doi: 10.19678/j.issn.1000-3428.0060733
24	ZHOU P, HAN X T, MORARIU V I, et al. Learning rich features for image manipulation detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 1053-1061.
25	FRIDRICH J, KODOVSKY J. Rich models for steganalysis of digital images. IEEE Transactions on Information Forensics and Security, 2012, 7 (3): 868- 882. doi: 10.1109/TIFS.2012.2190402
26	LUO Y C, ZHANG Y, YAN J C, et al. Generalizing face forgery detection with high-frequency features[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 16312-16321.
27	SUN K, LIU H, YE Q X, et al. Domain general face forgery detection by learning to weight. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35 (3): 2638- 2646. doi: 10.1609/aaai.v35i3.16367
28	马欣, 吉立新, 李邵梅. 基于多尺度Transformer融合多域信息的伪造人脸检测. 计算机科学, 2023, 50 (10): 112- 118. doi: 10.11896/jsjkx.220900048
	MA X, JI L X, LI S M. Forgery face detection based on multi-scale transformer fusing multi-domain information. Computer Science, 2023, 50 (10): 112- 118. doi: 10.11896/jsjkx.220900048
29	KING D E. Dlib-ml: a machine learning toolkit. Journal of Machine Learning Research, 2009, 10, 1755- 1758. doi: 10.1145/1577069.1755843
30	LI Y Z, YANG X, SUN P, et al. Celeb-DF: a new dataset for DeepFake forensics[EB/OL]. [2023-05-10]. https://arxiv.org/abs/1909.12962.
31	DOLHANSKY B, BITTON J, PFLAUM B, et al. The DeepFake Detection Challenge(DFDC) dataset[EB/OL]. [2023-05-10]. https://arxiv.org/pdf/2006.07397.
32	Google. Deepfake detecttion dataset[EB/OL]. (2019-09-24)[2023-05-10]. https://ai.googleblog.com/2019/09/contributing-data-to-deepfake-detection.html.
33	DENG J K, GUO J, VERVERAS E, et al. RetinaFace: single-shot multi-level face localisation in the wild[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 5203-5212.
34	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60 (6): 84- 90. doi: 10.1145/3065386
35	TAN M, LE Q. EfficientNet: rethinking model scaling for convolutional neural networks[C]//Proceedings of International Conference on Machine Learning. [S. l.]: PMLR, 2019: 6105-6114.
36	CHEN S, YAO T P, CHEN Y, et al. Local relation learning for face forgery detection. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35 (2): 1081- 1088. doi: 10.1609/aaai.v35i2.16193
37	YADAV D, SALMANI S. Deepfake: a survey on facial forgery technique using generative adversarial network[C]//Proceedings of the International Conference on Intelligent Computing and Control Systems. Washington D. C., USA: IEEE Press, 2019: 852-857.
38	TOLOSANA R, VERA-RODRIGUEZ R, FIERREZ J, et al. Deepfakes and beyond: a survey of face manipulation and fake detection. Information Fusion, 2020, 64, 131- 148. doi: 10.1016/j.inffus.2020.06.014
39	SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 2020, 128 (2): 336- 359. doi: 10.1007/s11263-019-01228-7

[1]	张天鹏, 韩晶, 吕学强. 基于多任务学习的超分辨率辅助小目标检测[J]. 计算机工程, 2024, 50(9): 304-312.
[2]	郭敏, 张熙涵, 李阳. 融合注意力的教师互一致性半监督医学图像分割[J]. 计算机工程, 2024, 50(9): 313-323.
[3]	曾钰琦, 刘博, 钟柏昌, 钟瑾. 智慧教育下基于改进YOLOv8的学生课堂行为检测算法[J]. 计算机工程, 2024, 50(9): 344-355.
[4]	李俊俊, 董建刚, 李坤. 基于Kubernetes的集群节能策略研究[J]. 计算机工程, 2024, 50(9): 82-91.
[5]	林畅, 郭伟, 任哲聪, 金海波. 基于Transformer的目标跟踪与分割统一算法[J]. 计算机工程, 2024, 50(9): 130-141.
[6]	李泽霖, 吕兆峰, 陈富强, 李克. 基于多跳信息融合的实体对齐模型[J]. 计算机工程, 2024, 50(9): 142-152.
[7]	王汝英, 马嘉骏, 董建强, 刘万龙, 张海涛, 尹凯, 赵博超. 基于MTS-BiGRU-DMHSA的工业负荷预测方法[J]. 计算机工程, 2024, 50(9): 169-178.
[8]	李俊仪, 李向阳, 龙朝勋, 李海燕, 李红松, 余鹏飞. 基于多级区域选择与跨层特征融合的野生菌分类[J]. 计算机工程, 2024, 50(9): 179-188.
[9]	朱凯, 李理, 张彤, 江晟, 别一鸣. 基于Transformer的多阶段运动模糊图像修复网络[J]. 计算机工程, 2024, 50(9): 276-285.
[10]	张华青, 夏张涛, 陆晓庆, 童基均. 基于字形特征的血管外科命名实体识别[J]. 计算机工程, 2024, 50(8): 13-21.
[11]	饶日昕, 王怡文, 曾砺志, 童心恬, 赵海涛. 面向废旧电缆检测的轻量化网络模型[J]. 计算机工程, 2024, 50(8): 22-30.
[12]	李华昱, 张智康, 闫阳, 岳阳. 基于知识图谱增强的领域多模态实体识别[J]. 计算机工程, 2024, 50(8): 31-39.
[13]	王蕾, 党时鹏, 潘丰. 基于卷积神经网络的隐匿性旁路预测模型[J]. 计算机工程, 2024, 50(8): 40-49.
[14]	陈瀚, 赵春蕾, 蒋昊达, 王春东. 基于融合模型与语义网络的App用户意图识别研究[J]. 计算机工程, 2024, 50(8): 50-63.
[15]	刘锁兰, 王炎, 王洪元, 朱生升. 基于多流语义图卷积网络的人体行为识别[J]. 计算机工程, 2024, 50(8): 64-74.

选择文件类型/文献管理软件名称

选择包含的内容