基于改进薄板样条运动模型的人脸动画算法

doi:10.19678/j.issn.1000-3428.0068130

摘要/Abstract

摘要： 面部动画在电影、游戏、虚拟现实等领域起着关键作用,对于实现逼真、生动的人脸动画和情感传达至关重要。当面临面部形状、姿态、表情等多个变化因素时,虽然通过薄板样条非线性变换可以获得较好的运动估计结果,但在处理面部复杂纹理和嘴部运动时存在运动估计不精细的问题,需要更强大的图像修复能力。因此,提出一种基于改进薄板样条运动模型(TPSMM)的人脸动画算法。首先,在TPSMM的基础上引入一种Farneback光流金字塔算法,通过与薄板样条变换和背景仿射变换相结合,使得人脸局部运动估计更精准;其次,为了更真实地恢复缺失区域的细节纹理信息,提出一种多尺度细节感知网络,该网络在编码器中通过嵌入通道注意力(ECA)模块减少源图像因多层下采样而导致的人脸细节信息丢失,在解码器中利用坐标注意力(CA)模块来有效捕获运动估计特征图中不同位置的重要特征,提高人脸图像的生成质量。实验结果表明,相比一阶段运动模型(FOMM)、关节动画的运动表示法(MRAA)、TPSMM等,该算法在MUG、UvA-Nemo和Oulu-CASIA数据集上的L1、平均关键点距离(AKD)、平均欧氏距离(AED)数值均达到最优,平均分别为0.0129、0.923、0.00099。

关键词: 面部动画, 光流估计, 薄板样条, 多尺度特征融合, 通道注意力机制, 坐标注意力机制

Abstract: Facial animation plays a crucial role in applications involving movies, games, and virtual reality in terms of achieving realistic and vivid emotional communication. When handling multiple factors, such as facial shape, posture, and expression, good motion estimation results can be obtained through thin plate spline nonlinear transformation. However, this approach results in imprecise motion estimation when dealing with complex facial textures and mouth movements, necessitating better image restoration capabilities. To address this issue, this paper proposes a facial animation algorithm based on an improved Thin Plate Spline Motion Model (TPSMM). First, based on TPSMM, a Farneback optical flow pyramid algorithm is introduced, which combines the thin plate spline and background affine transformations to enhance the accuracy of local facial motion estimation. Second, to accurately recover the detailed textural information for missing areas, a multi-scale detail perception network is introduced. This network minimizes the loss of facial detail information caused by multi-layer downsampling of the source image by Embedding Channel Attention (ECA) modules in the encoder. In the decoder, the Coordinate Attention (CA) module effectively captures important features at different positions in the motion estimation feature map, thereby improving the quality of facial image generation. Experimental results show that, compared to the First Order Motion Model (FOMM), Motion Representations for Articulated Animation (MRAA), and TPSMM, the proposed algorithm achieves optimal L1, Average Keypoint Distance (AKD), and Average Euclidean Distance (AED) values on the MUG, UvA-Nemo, and Oulu-CASIA datasets, with averages of 0.0129, 0.923, and 0.00099, respectively.

Key words: facial animation, optical flow estimation, thin plate spline, multi-scale feature fusion, channel attention mechanism, coordinate attention mechanism

中图分类号:

TP391

杨硕, 王一丁. 基于改进薄板样条运动模型的人脸动画算法[J]. 计算机工程, 2024, 50(6): 255-265.

YANG Shuo, WANG Yiding. Facial Animation Algorithm Based on Improved Thin Plate Spline Motion Model[J]. Computer Engineering, 2024, 50(6): 255-265.

https://www.ecice06.com/CN/Y2024/V50/I6/255

参考文献

[1] CHOPRA A, JAIN R, HEMANI M, et al. ZFlow:gated appearance flow-based virtual try-on with 3D priors[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D.C.,USA:IEEE Press,2021:5433-5442.
[2] BHATNAGAR B, TIWARI G, THEOBALT C, et al. Multi-garment net:learning to dress 3D people from images[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D.C.,USA:IEEE Press,2019:5419-5429.
[3] WANG T C, MALLYA A, LIU M Y. One-shot free-view neural talking-head synthesis for video conferencing[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C.,USA:IEEE Press,2021:10039-10049.
[4] NARUNIEC J, HELMINGER L, SCHROERS C, et al. High-resolution neural face swapping for visual effects[J]. Computer Graphics Forum, 2020, 39(4):173-184.
[5] MA L, JIA X, SUN Q, et al. Pose guided person image generation[EB/OL].[2023-06-05].https://arxiv.org/abs/1705.09368.
[6] MA L Q, SUN Q R, GEORGOULIS S, et al. Disentangled person image generation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C.,USA:IEEE Press,2018:99-108.
[7] NI H M, LIU Y H, HUANG S X, et al. Cross-identity video motion retargeting with joint transformation and synthesis[C]//Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision. Washington D.C.,USA:IEEE Press, 2023:412-422.
[8] ZHAO L, PENG X, TIAN Y, et al. Learning to forecast and refine residual motion for image-to-video generation[C]//Proceedings of European Conference on Computer Vision. Berlin,Germany:Springer,2018:403-419.
[9] REN Y R, LI G, CHEN Y Q, et al. PIRenderer:controllable portrait image generation via semantic neural rendering[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D.C.,USA:IEEE Press,2021:13759-13768.
[10] LIU W, PIAO Z X, MIN J, et al. Liquid warping GAN:a unified framework for human motion imitation, appearance transfer and novel view synthesis[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D.C.,USA:IEEE Press,2019:5903-5912.
[11] PUMAROLA A, AGUDO A, MARTINEZ A M, et al. GANimation:anatomically-aware facial animation from a single image[C]//Proceedings of the 15th European Conference on Computer Vision. New York,USA:ACM Press,2018:835-851.
[12] EKMAN P, ROSENBERG E L. What the face reveals:basic and applied studies of spontaneous expression using the facial action coding system (FACS)[M]. 2nd ed. Oxford, UK:Oxford University Press, 2005.
[13] SIAROHIN A, LATHUILIERE S, TULYAKOV S, et al. Animating arbitrary objects via deep motion transfer[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C.,USA:IEEE Press,2019:2377-2386.
[14] TAO J L, WANG B, XU B R, et al. Structure-aware motion transfer with deformable anchor model[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C.,USA:IEEE Press,2022:3627-3636.
[15] TAO J L, WANG B, GE T Z, et al. Motion transformer for unsupervised image animation[EB/OL].[2023-06-05]. https://arxiv.org/abs/2209.14024.
[16] SIAROHIN A, LATHUILIōRE S, TULYAKOV S, et al. First order motion model for image animation[EB/OL].[2023-06-05]. https://arxiv.org/abs/2003.00196.
[17] SIAROHIN A, WOODFORD O J, REN J, et al. Motion representations for articulated animation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C.,USA:IEEE Press,2021:13648-13657.
[18] ZHAO J, ZHANG H. Thin-plate spline motion model for image animation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C.,USA:IEEE Press,2022:3657-3666.
[19] FARNEBÄCK G. Two-frame motion estimation based on polynomial expansion[EB/OL].[2023-06-05]. https://link.springer.com/chapter/10.1007/3-540-45103-X_50.
[20] BOOKSTEIN F L. Principal warps:thin-plate splines and the decomposition of deformations[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1989, 11(6):567-585.
[21] RONNEBERGER O, FISCHER P, BROX T. U-Net:convolutional networks for biomedical image segmentation[EB/OL].[2023-06-05]. https://arxiv.org/abs/1505.04597.
[22] BULAT A, TZIMIROPOULOS G. How far are we from solving the 2D&3D face alignment problem?(and a dataset of 230, 0003D facial landmarks)[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D.C.,USA:IEEE Press,2017:1021-1030.
[23] LIANG J, ZENG H, ZHANG L. High-resolution photorealistic image translation in real-time:a Laplacian pyramid translation network[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C.,USA:IEEE Press,2021:9392-9400.
[24] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C.,USA:IEEE Press,2016:770-778.
[25] 兰朝凤,蒋朋威,陈欢,等.结合光流算法与注意力机制的U-Net网络跨模态视听语音分离[J].电子与信息学报, 2023, 45(10):3538-3546. LAN C F, JIANG P W, CHEN H, et al. Cross-modal audiovisual separation based on U-Net network combining optical flow algorithm and attention mechanism[J]. Journal of Electronics&Information Technology, 2023, 45(10):3538-3546.(in Chinese)
[26] HEITZ D, MÉMIN E, SCHNÖRR C. Variational fluid flow measurements from image sequences:synopsis and perspectives[J]. Experiments in Fluids, 2010, 48(3):369-393.
[27] WANG Q L, WU B G, ZHU P F, et al. ECA-Net:efficient channel attention for deep convolutional neural networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C.,USA:IEEE Press,2020:11534-11542.
[28] 熊中敏,曾旗,卢鹏,等.基于残差注意力多尺度关系网络的逻辑推理[J].计算机工程, 2023, 49(6):227-233, 241. XIONG Z M, ZENG Q, LU P, et al. Logical reasoning based on residual attention multi-scale relation network[J]. Computer Engineering, 2023, 49(6):227-233, 241.(in Chinese)
[29] HOU Q B, ZHOU D Q, FENG J S. Coordinate attention for efficient mobile network design[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C.,USA:IEEE Press,2021:13713-13722.
[30] 孙伟,常鹏帅,戴亮,等.基于注意力引导数据增强的车型识别[J].计算机工程, 2022, 48(7):300-306. SUN W, CHANG P S, DAI L, et al. Vehicle type recognition based on attention guided data augmentation[J]. Computer Engineering, 2022, 48(7):300-306.(in Chinese)
[31] JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of European Conference on Computer Vision. Berlin,Germany:Springer,2016:694-711.
[32] AIFANTI N, PAPACHRISTOU C, DELOPOULOS A. The mug facial expression database[EB/OL].[2023-06-05]. https://mug.ee.auth.gr/fed/.
[33] DIBEKLIOĞLU H, ALI SALAH A, GEVERS T. Are you really smiling at me?spontaneous versus posed enjoyment smiles[EB/OL].[2023-06-05]. https://link.springer.com/content/pdf/10.1007/978-3-642-33712-3_38.
[34] ZHAO G Y, HUANG X H, TAINI M, et al. Facial expression recognition from near-infrared videos[J]. Image and Vision Computing, 2011, 29(9):607-619.
[35] WANG W, ALAMEDA-PINEDA X, XU D, et al. Every smile is unique:landmark-guided diverse smile generation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C.,USA:IEEE Press,2018:7083-7092.
[36] AMOS B, LUDWICZUK B, SATYANARAYANAN M. OpenFace:a general-purpose face recognition library with mobile applications[EB/OL].[2023-06-05].https://elijah.cs.cmu.edu/DOCS/CMU-CS-16-118.pdf.
[37] ALLEN D M. Mean square error of prediction as a criterion for selecting variables[J]. Technometrics, 1971, 13(3):469-475.

选择文件类型/文献管理软件名称

选择包含的内容