人脸微表情分析方法综述

doi:10.19678/j.issn.1000-3428.0065790

摘要/Abstract

摘要： 微表情分析在医学、公共安全、商业谈判等领域得到广泛应用并备受关注。微表情运动幅度小、变化快，导致人工分析难度较大，开发一个可靠的自动化微表情分析系统非常有必要。随着计算机视觉技术的发展，研究人员能够结合相关算法捕捉微表情运动变化特征以用于微表情分析。阐述微表情分析的发展历程和现状，从多个角度对微表情分析的两大分支，即微表情检测方法和微表情识别方法进行总结。整理现有微表情数据集以及微表情分析流程中常用的面部图像预处理方法。根据特征提取方式的不同，从基于时间特征、基于特征变化和基于深度特征这3个方面对微表情检测方法进行阐述。将微表情识别方法归纳为基于纹理特征和基于光流特征的传统机器学习方法以及深度学习方法，其中，基于深度学习的微表情识别包括基于运动单元、基于关键帧和基于迁移学习的方法。通过不同实验指标对以上方法进行分析和比较，在此基础上，探讨当前微表情分析中存在的问题和挑战并展望该领域未来的发展方向。

关键词: 微表情分析, 计算机视觉, 微表情检测, 微表情识别, 深度学习

Abstract: Micro-expression analysis has been widely used in the medical industry, public safety, business disputes and other fields and has attracted much attention.Manual analysis is challenging because the intensity of micro-expressions is relatively modest and they fluctuate quickly.Therefore, automated methods must be developed to perform micro-expression analysis.Driven by advancements in learning models, many works on computer vision have considered the use of subtle facial movements to investigate micro-expressions.This study thoroughly describes the evolution of micro-expression analysis.The two key issues involved, including detection and identification methods, are summarized rigorously from various perspectives in light of the current state of the art.First, existing micro-expression datasets and facial image preprocessing methods that are commonly used in micro-expression analysis are summarized and introduced.Then, according to different feature extraction methods, micro-expression detection methods are divided into those that consider temporal features, feature change, and depth features.Micro-expression recognition techniques are largely based on texture features, optical flow features of traditional machine learning methods, and deep learning methods.Deep learning methods have been developed for micro-expression recognition based on motion units, key frames, and transfer learning methods.Thus, several experimental indicators are analyzed and compared.In conclusion, the current problems and challenges in micro-expression analysis are discussed and some possible directions for future development are suggested.

Key words: micro-expression analysis, computer vision, micro-expression detection, micro-expression recognition, deep learning

中图分类号:

TP391

于明, 钟元想, 王岩. 人脸微表情分析方法综述[J]. 计算机工程, 2023, 49(2): 1-14.

YU Ming, ZHONG Yuanxiang, WANG Yan. A Survey of Facial Micro-Expression Analysis Methods[J]. Computer Engineering, 2023, 49(2): 1-14.

http://www.ecice06.com/CN/Y2023/V49/I2/1

图/表 14

20230216175447

20230216175451

20230216175455

20230216175458

20230216175502

20230216175506

20230216175510

20230216175514

20230216175517

20230216175523

20230216175527

20230216175532

20230216175537

20230216175542

参考文献

[1] EKMAN P, FRIESEN W V.Nonverbal leakage and clues to deception[J].Psychiatry, 1969, 32(1):88-106.
[2] WHITEHILL J, SERPELL Z, LIN Y C, et al.The faces of engagement:automatic recognition of student engagementfrom facial expressions[J].IEEE Transactions on Affective Computing, 2014, 5(1):86-98.
[3] EKMAN P, O'SULLIVAN M.From flawed self-assessment to blatant whoppers:the utility of voluntary and involuntary behavior in detecting deception[J].Behavioral Sciences & the Law, 2006, 24(5):673-686.
[4] ZHANG M, FU Q, CHEN Y H, et al.Emotional context influences micro-expression recognition[J].PLoS One, 2014, 9(4):e95018.
[5] MOILANEN A, ZHAO G Y, PIETIKÄINEN M.Spotting rapid facial movements from videos using appearance-based feature difference analysis[C]//Proceedings of the 22nd International Conference on Pattern Recognition.Washington D.C., USA:IEEE Press, 2014:1722-1727.
[6] HUANG X H, WANG S J, ZHAO G Y, et al.Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection[C]//Proceedings of IEEE International Conference on Computer Vision Workshop.Washington D.C., USA:IEEE Press, 2015:1-9.
[7] HUANG X H.Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[J].Neurocomputing, 2016, 175:564-578.
[8] LIU Y J, ZHANG J K, YAN W J, et al.A main directional mean optical flow feature for spontaneous micro-expression recognition[J].IEEE Transactions on Affective Computing, 2016, 7(4):299-310.
[9] WANG S J, YAN W J, LI X B, et al.Micro-expression recognition using color spaces[J].IEEE Transactions on Image Processing, 2015, 24(12):6034-6047.
[10] BEN X Y, ZHANG P, YAN R, et al.Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation[J].Neural Computing and Applications, 2016, 27(8):2629-2646.
[11] KHOR H Q, SEE J, PHAN R C W, et al.Enriched long-term recurrent convolutional network for facial micro-expression recognition[C]//Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition.Washington D.C., USA:IEEE Press, 2018:667-674.
[12] LO L, XIE H X, SHUAI H H, et al.MER-GCN:micro-expression recognition based on relation modeling with graph convolutional networks[C]//Proceedings of IEEE Conference on Multimedia Information Processing and Retrieval.Washington D.C., USA:IEEE Press, 2020:79-84.
[13] ZHOU Y, SONG Y, CHEN L, et al.A novel micro-expression detection algorithm based on BERT and 3DCNN[J].Image and Vision Computing, 2022, 119:104378.
[14] SHREVE M, GODAVARTHY S, GOLDGOF D, et al.Macro-and micro-expression spotting in long videos using spatio-temporal strain[C]//Proceedings of IEEE International Conference on Automatic Face & Gesture Recognition.Washington D.C., USA:IEEE Press, 2011:51-56.
[15] POLIKOVSKY S, KAMEDA Y, OHTA Y.Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor[J].Journal of Nonverbal Behavior, 2009, 33(1):59-69.
[16] LI X B, PFISTER T, HUANG X H, et al.A spontaneous micro-expression database:inducement, collection and baseline[C]//Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition.Washington D.C., USA:IEEE Press, 2013:1-6.
[17] ASTHANA A, ZAFEIRIOU S, CHENG S Y, et al.Robust discriminative response map fitting with constrained local models[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2013:3444-3451.
[18] YAN W J, LI X, WANG S J, et al.CASMEII:an improved spontaneous micro-expression database and the baseline evaluation[J].PLoS One, 2014, 9(1):e86041.
[19] QU F B, WANG S J, YAN W J, et al.CAS(ME)²:a database for spontaneous macro-expression and micro-expression spotting and recognition[J].IEEE Transactions on Affective Computing, 2018, 9(4):424-436.
[20] LI J T, DONG Z Z, LU S Y, et al.CAS(ME)³:a third generation facial spontaneous micro-expression database with depth information and high ecological validity[EB/OL].[2022-08-05].https://pubmed.ncbi.nlm.nih.gov/35560102/.
[21] DAVISON A K, LANSLEY C, COSTEN N, et al.SAMM:a spontaneous micro-facial movement dataset[J].IEEE Transactions on Affective Computing, 2018, 9(1):116-129.
[22] BEN X Y, REN Y, ZHANG J P, et al.Video-based facial micro-expression analysis:a survey of datasets, features and algorithms[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(9):5826-5846.
[23] WARREN G, SCHERTLER E, BULL P.Detecting deception from emotional and unemotional cues[J].Journal of Nonverbal Behavior, 2009, 33(1):59-69.
[24] DADI H S, MOHAN PILLUTLA G K.Improved face recognition rate using HOG features and SVM classifier[J].IOSR Journal of Electronics and Communication Engineering, 2016, 11(4):34-44.
[25] LIN T Y, DOLLÁR P, GIRSHICK R, et al.Feature pyramid networks for object detection[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:936-944.
[26] COOTES T F.Active shape models-their training and application[J].Computer Vision and Image Understanding, 1995, 61(1):38-59.
[27] LUCEY S, WANG Y, COX M, et al.Efficient constrained local model fitting for non-rigid face alignment[J].Image and Vision Computing, 2009, 27(12):1804-1813.
[28] MERGHANI W, YAP M H.Adaptive mask for region-based facial micro-expression recognition[C]//Proceedings of 2020 IEEE International Conference on Automatic Face and Gesture Recognition.Washington D.C., USA:IEEE Press, 2020:765-770.
[29] SHREVE M, GODAVARTHY S, MANOHAR V, et al.Towards macro- and micro-expression spotting in video using strain patterns[C]//Proceedings of Workshop on Applications of Computer Vision.Washington D.C., USA:IEEE Press, 2009:1-6.
[30] PATEL D, ZHAO G, PIETIK INEN M.Spatiotemporal integration of optical flow vectors for micro-expression detection[C]//Proceedings of International Conference on Advanced Concepts for Intelligent Vision Systems.Berlin, Germany:Springer, 2015:369-380.
[31] MA H Y, AN G Y, WU S J, et al.A Region Histogram of Oriented Optical Flow(RHOOF) feature for apex frame spotting in micro-expression[C]//Proceedings of International Symposium on Intelligent Signal Processing and Communication Systems.Washington D.C., USA:IEEE Press, 2017:281-286.
[32] GUO Y F, LI B, BEN X Y, et al.A magnitude and angle combined optical flow feature for microexpression spotting[J].IEEE MultiMedia, 2021, 28(2):29-39.
[33] WANG S J.A main directional maximal difference analysis for spotting facial movements from long-term videos[J].Neurocomputing, 2017, 230:382-389.
[34] HE Y, WANG S J, LI J T, et al.Spotting macro- and micro- expression intervals in long video sequences[C]//Proceedings of the 15th IEEE International Conference on Automatic Face and Gesture Recognition.Washington D.C., USA:IEEE Press, 2020:742-748.
[35] ZHANG L W, LI J T, WANG S J, et al.Spatio-temporal fusion for macro- and micro-expression spotting in long video sequences[C]//Proceedings of the 15th IEEE International Conference on Automatic Face and Gesture Recognition.Washington D.C., USA:IEEE Press, 2020:734-741.
[36] HE Y, XU Z, MA L, et al.Micro-expression spotting based on optical flow features[J].Pattern Recognition Letters, 2022, 163:57-64.
[37] POLIKOVSKY S, KAMEDA Y, OHTA Y.Detection and measurement of facial micro-expression characteristics for psychological analysis[J].Kameda's Publication, 2010, 110:57-64.
[38] DAVISON A K, YAP M H, LANSLEY C.Micro-facial movement detection using individualised baselines and histogram-based descriptors[C]//Proceedings of 2015 IEEE International Conference on Systems, Man, and Cybernetics.Washington D.C., USA:IEEE Press, 2015:1864-1869.
[39] XIA Z Q.Spontaneous micro-expression spotting via geometric deformation modeling[J].Computer Vision and Image Understanding, 2016, 147:87-94.
[40] LI X B, HONG X P, MOILANEN A, et al.Towards reading hidden emotions:a comparative study of spontaneous micro-expression spotting and recognition methods[J].IEEE Transactions on Affective Computing, 2018, 9(4):563-577.
[41] YAN W J, WANG S J, CHEN Y H, et al.Quantifying micro-expressions with constraint local model and local binary pattern[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2015:296-305.
[42] LIONG S T, SEE J, WONG K, et al.Automatic apex frame spotting in micro-expression database[C]//Proceedings of the 3rd IAPR Asian Conference on Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:665-669.
[43] HAN Y H, LI B J, LAI Y K, et al.CFD:a collaborative feature difference method for spontaneous micro-expression spotting[C]//Proceedings of the 25th IEEE International Conference on Image Processing.Washington D.C., USA:IEEE Press, 2018:1942-1946.
[44] ZHANG Z H, CHEN T, MENG H Y, et al.SMEConvNet:a convolutional neural network for spotting spontaneous facial micro-expression from long videos[J].IEEE Access, 2018, 6:71143-71151.
[45] TRAN T K, VO Q N, HONG X P, et al.Dense prediction for micro-expression spotting based on deep sequence model[J].Electronic Imaging, 2019, 31(8):1-6.
[46] DING J, TIAN Z, LÜ X, et al.Real-time micro-expression detection in unlabeled long videos using optical flow and LSTM neural network[C]//Proceedings of International Conference on Computer Analysis of Images and Patterns.Berlin, Germany:Springer, 2019:622-634.
[47] PAN H, XIE L, WANG Z L.Local bilinear convolutional neural network for spotting macro- and micro-expression intervals in long video sequences[C]//Proceedings of 2020 IEEE International Conference on Automatic Face and Gesture Recognition.Washington D.C., USA:IEEE Press, 2020:749-753.
[48] LIONG G B, SEE J, WONG L K.Shallow optical flow three-stream CNN for macro- and micro-expression spotting from long videos[C]//Proceedings of IEEE International Conference on Image Processing.Washington D.C., USA:IEEE Press, 2021:2643-2647.
[49] WANG S J, HE Y, LI J T, et al.MESNet:a convolutional neural network for spotting multi-scale micro-expression intervals in long videos[J].IEEE Transactions on Image Processing, 2021, 30:3956-3969.
[50] XUE L Y, ZHU T, HAO J S.A two-stage deep neural network for macro- and micro-expression spotting from long-term videos[C]//Proceedings of the 14th International Symposium on Computational Intelligence and Design.Washington D.C., USA:IEEE Press, 2021:282-286.
[51] TAKALKAR M A.LGAttNet:automatic micro-expression detection using dual-stream local and global attentions[J].Knowledge-Based Systems, 2021, 212:106566.
[52] GUPTA P.PERSIST:improving micro-expression spotting using better feature encodings and multi-scale Gaussian TCN[J].Applied Intelligence, 2022, 10:1-15.
[53] PFISTER T, LI X B, ZHAO G Y, et al.Recognising spontaneous facial micro-expressions[C]//Proceedings of International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2011:1449-1456.
[54] WANG Y D, SEE J, PHAN R C W, et al.LBP with six intersection points:reducing redundant information in LBP-TOP for micro-expression recognition[C]//Proceedings of Asian Conference on Computer Vision.Berlin, Germany:Springer, 2015:525-537.
[55] HUANG X H, WANG S J, LIU X, et al.Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition[J].IEEE Transactions on Affective Computing, 2019, 10(1):32-47.
[56] WEI J S, LU G M, YAN J J, et al.Micro-expression recognition using local binary pattern from five intersecting planes[J].Multimedia Tools and Applications, 2022, 81(15):20643-20668.
[57] WEI J, LU G, YAN J.A comparative study on movement feature in different directions for micro-expression recognition[J].Neurocomputing, 2021, 449:159-171.
[58] LIU Y J, LI B J, LAI Y K.Sparse MDMO:learning a discriminative feature for micro-expression recognition[J].IEEE Transactions on Affective Computing, 2021, 12(1):254-261.
[59] LIONG S T.Less is more:micro-expression recognition from video using apex frame[J].Signal Processing:Image Communication, 2018, 62:82-92.
[60] HAPPY S L, ROUTRAY A.Fuzzy histogram of optical flow orientations for micro-expression recognition[J].IEEE Transactions on Affective Computing, 2019, 10(3):394-406.
[61] WANG L.A weighted feature extraction method based on temporal accumulation of optical flow for micro-expression recognition[J].Signal Processing:Image Communication, 2019, 78:246-253.
[62] LEI L, LI J F, CHEN T, et al.A novel Graph-TCN with a graph structured representation for micro-expression recognition[C]//Proceedings of the 28th ACM International Conference on Multimedia.New York, USA:ACM Press, 2020:2237-2245.
[63] LEI L, CHEN T, LI S G, et al.Micro-expression recognition based on facial graph representation learning and facial action unit fusion[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2021:1571-1580.
[64] ZHAO X, MA H, WANG R.STA-GCN:spatio-temporal AU graph convolution network for facial micro-expression recognition[C]//Proceedings of Chinese Conference on Pattern Recognition and Computer Vision.Berlin, Germany:Springer, 2021:80-91.
[65] SUN X, LANG J J, ZHENG S X, et al.Dual expression fusion:a universal microexpression recognition framework[EB/OL].[2022-08-05].https://www.computer.org/csdl/magazine/mu/5555/01/09817631/1EOA8CImZYQ.
[66] WANG Y, ZHENG S X, SUN X, et al.Micro-expression recognition with attention mechanism and region enhancement[EB/OL].[2022-08-05].https://link.springer.com/article/10.1007/s00530-022-00934-6.
[67] CEN S, YU Y, YAN G, et al.Multi-task facial activity patterns learning for micro-expression recognition using joint temporal local cube binary pattern[J].Signal Processing:Image Communication, 2022, 103:116616.
[68] ZHOU L, MAO Q R, XUE L Y.Cross-database micro-expression recognition:a style aggregated and attention transfer approach[C]//Proceedings of IEEE International Conference on Multimedia & Expo Workshops.Washington D.C., USA:IEEE Press, 2019:102-107.
[69] LI Y T, HUANG X H, ZHAO G Y.Joint local and global information learning with single apex frame detection for micro-expression recognition[J].IEEE Transactions on Image Processing, 2021, 30:249-263.
[70] LIONG S T, GAN Y S, SEE J, et al.Shallow Triple Stream Three-dimensional CNN (STSTNet) for micro-expression recognition[C]//Proceedings of the 14th IEEE International Conference on Automatic Face & Gesture Recognition.Washington D.C., USA:IEEE Press, 2019:1-5.
[71] ZHAO Y, XU J.A convolutional neural network for compound micro-expression recognition[J].Sensors (Basel), 2019, 19(24):E5553.
[72] 刘德志, 梁正友, 孙宇.结合空间注意力机制与光流特征的微表情识别方法[J].计算机辅助设计与图形学学报, 2021, 33(10):1541-1552. LIU D Z, LIANG Z Y, SUN Y.Micro-expression recognition method based on spatial attention mechanism and optical flow features[J].Journal of Computer-Aided Design & Computer Graphics, 2021, 33(10):1541-1552.(in Chinese)
[73] OH T H, JAROENSRI R, KIM C, et al.Learning-based video motion magnification[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2018:633-648.
[74] GAN Y S.OFF-ApexNet on micro-expression recognition system[J].Signal Processing:Image Communication, 2019, 74:129-139.
[75] XIA Z Q, PENG W, KHOR H Q, et al.Revealing the invisible with model and data shrinking for composite-database micro-expression recognition[J].IEEE Transactions on Image Processing, 2020, 29:8590-8605.
[76] LI X Y, WEI G S, WANG J, et al.Multi-scale joint feature network for micro-expression recognition[J].Computational Visual Media, 2021, 7(3):407-417.
[77] CHEN B, LIU K H, XU Y, et al.Block division convolutional network with implicit deep features augmentation for micro-expression recognition[EB/OL].[2022-08-05].https://ieeexplore.ieee.org/document/9676412.
[78] 梁正友, 刘德志, 孙宇.结合迁移学习与可分离三维卷积的微表情识别方法[J].计算机工程, 2022, 48(1):228-235. LIANG Z Y, LIU D Z, SUN Y.Micro-expression recognition method combining transfer learning and separable 3D convolution[J].Computer Engineering, 2022, 48(1):228-235.(in Chinese)
[79] BEN X Y.Learning effective binary descriptors for micro-expression recognition transferred by macro-information[J].Pattern Recognition Letters, 2018, 107:50-58.
[80] XIA B, WANG W K, WANG S F, et al.Learning from macro-expression:a micro-expression recognition framework[C]//Proceedings of the 28th ACM International Conference on Multimedia.New York, USA:ACM Press, 2020:2936-2944.
[81] WANG C Y.Micro-attention for micro-expression recognition[J].Neurocomputing, 2020, 410:354-362.
[82] WEBER R, LI J, SOLADI C, et al.A survey on databases of facial macro-expression and micro-expression[C]//Proceedings of International Joint Conference on Computer Vision, Imaging and Computer Graphics.Berlin, Germany:Springer, 2018:298-325.
[83] TANG H, CHAI L, LU W.Transferring dual stochastic graph convolutional network for facial micro-expression recognition[EB/OL].[2022-08-05].https://doi.org/10.48550/arXiv.2203.05208.
[84] ZHANG J, YAN B Y, DU X H, et al.Motion magnification multi-feature relation network for facial microexpression recognition[J].Complex & Intelligent Systems, 2022, 8(4):3363-3376.

选择文件类型/文献管理软件名称

选择包含的内容