结合迁移学习与可分离三维卷积的微表情识别方法

doi:10.19678/j.issn.1000-3428.0059771

摘要/Abstract

摘要： 针对现有微表情自动识别方法准确率较低及微表情样本数量不足的问题，提出一种融合迁移学习技术与可分离三维卷积神经网络（S3D CNN）的微表情识别方法。通过光流法提取宏表情和微表情视频样本的光流特征帧序列，利用宏表情样本的光流特征帧序列对S3D CNN进行预训练，并采用微表情样本的光流特征帧序列微调模型参数。S3D CNN网络由二维空域卷积层及添加一维时域卷积层的可分离三维卷积层构成，比传统的三维卷积神经网络具有更好的学习能力，且减少了模型所需的训练参数和计算量。在此基础上，采用迁移学习的方式对模型进行训练，以缓解微表情样本数量过少造成的模型过拟合问题，提升模型的学习效率。实验结果表明，所提方法在CASME II微表情数据集上的识别准确率为67.58%，高于MagGA、C3DEvol等前沿的微表情识别算法。

关键词: 微表情识别, 深度学习, 卷积神经网络, 迁移学习, 光流法

Abstract: The existing automatic micro-expression recognition methods are limited in accuracy, and suffer from inadequate micro-expression samples.To address the problem, a micro-expression recognition method that combines transfer learning and a Separable 3D Convolutional Neural Network(S3D CNN) is proposed.The optical flow method is used to extract the feature frame sequences of optical flow from macro-expression and micro-expression video samples. The sequence extracted from macro-expression samples is used to pre-train the S3D CNN, and the sequence extracted from micro-expression samples is used to tune the model parameters.S3D CNN consists of separable 3D convolutional layers, which are composed by 2D spatial convolutional layers and 1D time-domain convolutional layers, so S3D CNN can provide better learning ability than traditional 3D CNN with fewer required parameters and calculations for model training.Furthermore, transfer learning is used to train the model, so the over-fitting problem of the model caused by inadequate micro-expression samples can be alleviated, and the learning efficiency of the model can be improved. Experimental results on the CASME II micro-expression dataset show that the recognition accuracy of the proposed method reaches 67.58%, higher than MagGA, C3DEvol and other advanced algorithms.

Key words: micro-expression recognition, deep learning, convolutional neural networks, transfer learning, optical flow method

中图分类号:

TP391.41

梁正友, 刘德志, 孙宇. 结合迁移学习与可分离三维卷积的微表情识别方法[J]. 计算机工程, 2022, 48(1): 228-235.

LIANG Zhengyou, LIU Dezhi, SUN Yu. Micro-Expression Recognition Method Combining Transfer Learning and Separable 3D Convolution[J]. Computer Engineering, 2022, 48(1): 228-235.

http://www.ecice06.com/CN/Y2022/V48/I1/228

图/表 9

20220108123423

20220108123426

20220108123430

20220108123433

20220108123437

20220108123440

20220108123444

20220108123447

20220108123450

参考文献

[1] KHARAT G U, DUDUL S V.Emotion recognition from facial expression using neural networks[J].Springer, 2009, 60(3):207-219.
[2] YAN W J, WU Q, LIANG J, et al.How fast are the leaked facial expressions:the duration of micro-expressions[J].Journal of Nonverbal Behavior, 2013, 37(4):217-230.
[3] CARREIRA J, ZISSERMAN A.Quo vadis, action recognition?a new model and the kinetics dataset[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:6299-6308.
[4] TRAN D, BOURDEV L, FERGUS R, et al.Learning spatiotemporal features with 3D convolutional networks[C]//Proceedings of 2015 IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2015:4489-4497.
[5] LI J, WANG Y D, SEE J, et al.Micro-expression recognition based on 3D flow convolutional neural network[J].Pattern Analysis and Applications, 2019, 22(4):1331-1339.
[6] REDDY S P T, KARRI S T, DUBEY S R, et al.Spontaneous facial micro-expression recognition using 3D spatiotemporal convolutional neural networks[C]//Proceedings of 2019 International Joint Conference on Neural Networks.Washington D.C., USA:IEEE Press, 2019:1-8.
[7] PENG M, WANG C Y, CHEN T, et al.Dual temporal scale convolutional neural network for micro-expression recognition[J].Frontiers in Psychology, 2017, 8(3):1745-1745.
[8] HUANG X H, ZHAO G Y, HONG X P, et al.Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[J].Neurocomputing, 2016, 175(12):564-578.
[9] LIU Y J, ZHANG J K, YAN W J, et al.A main directional mean optical flow feature for spontaneous micro-expression recognition[J].IEEE Transactions on Affective Computing, 2016, 7(4):299-310.
[10] 梁正友, 何景琳, 孙宇.一种用于微表情自动识别的三维卷积神经网络进化方法[J].计算机科学, 2020, 47(8):227-232. LIANG Z Y, HE J L, SUN Y.Three-dimensional convolutional neural network evolution method for facial micro-expression auto recognition[J].Computer Science, 2020, 47(8):227-232.(in Chinese)
[11] WEISS K, KHOSHGOFTAAR T M, WANG D D.A survey of transfer learning[J].Journal of Big Data, 2016, 3(1):1-40.
[12] PATEL D, HONG X P, ZHAO G Y.Selective deep features for micro-expression recognition[C]//Proceedings of the 23rd International Conference on Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:2258-2263.
[13] PENG M, WU Z, ZHANG Z H, et al.From macro to micro expression recognition:deep learning on small datasets using transfer learning[C]//Proceedings of the13th IEEE International Conference on Automatic Face & Gesture Recognition.Washington D.C., USA:IEEE Press, 2018:657-661.
[14] LI Y T, HUANG X H, ZHAO G Y.Can micro-expression be recognized based on single apex frame?[C]//Proceedings of the 25th IEEE International Conference on Image Processing.Washington D.C., USA:IEEE Press, 2018:3094-3098.
[15] WU H Y, RUBINSTEIN M, SHIH E, et al.Eulerian video magnification for revealing subtle changes in the world[J].ACM Transactions on Graphics, 2012, 31(4):1-8.
[16] TRAN D, WANG H, TORRESANI L, et al.A closer look at spatiotemporal convolutions for action recognition[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:6450-6459.
[17] QIU Z F, YAO T, MEI T.Learning spatio-temporal representation with pseudo-3D residual networks[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2017:5533-5541.
[18] XIE S N, SUN C, HUANG JONATHAN, et al.Rethinking spatiotemporal feature learning:speed-accuracy trade-offs in video classification[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2018:318-335.
[19] ZHOU Z H, ZHAO G Y, PIETIKÄINEN M.Towards a practical lipreading system[C]//Proceedings of 2011 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2011:137-144.
[20] LIONG S T, GAN Y S, ZHENG D N, et al.Evaluation of the spatio-temporal features and GAN for micro-expression recognition system[J].Journal of Signal Processing Systems, 2020, 23(3):1-21.
[21] ZACH C, POCK T, BISCHOF H.A duality based approach for realtime TV-L1 optical flow[C]///Proceedings of Joint Pattern Recognition Symposium.Berlin, Germany:Springer, 2007:214-223.
[22] SHREVE M, GODAVARTHY S, MANOHAR V, et al.Towards macro-and micro-expression spotting in video using strain patterns[C]//Proceedings of Workshop on Applications of Computer Vision.Washington D.C., USA:IEEE Press, 2009:1-6.
[23] LIONG S T, SEE J, PHAN R C, et al.Subtle expression recognition using optical strain weighted features[C]//Proceedings of the 12th Asian Conference on Computer Vision.Berlin, Germany:Springer, 2014:644-657.
[24] LUCEY P, COHN J F, KANADE T, et al.The extended Cohn-Kanade dataset (CK+):a complete dataset for action unit and emotion-specified expression[C]//Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2010:94-10.
[25] YAN W J, LI X B, WANG S J, et al.CASME II:an improved spontaneous micro-expression database and the baseline evaluation[J].Plos One, 2014, 9(4):1-8.
[26] HUANG X H, WANG S J, ZHAO G Y, et al.Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection[C]//Proceedings of 2015 IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2015:1-9.

选择文件类型/文献管理软件名称

选择包含的内容