基于多尺度时空注意力网络的微表情检测方法

doi:10.19678/j.issn.1000-3428.0068016

摘要/Abstract

摘要： 微表情可以揭示人们试图隐藏的真实情绪,为刑事侦查、心理辅导等提供潜在的信息。现有微表情检测方法主要在获取空间特征的基础上提取时间特征以构建时空特征,这种处理方式容易导致时间特征失真,同时在空间处理过程中会破坏原有时序关系,降低微表情时空特征的判别性。针对这一问题,提出基于多尺度时空注意力网络的微表情检测方法。利用包含时间和空间关系的三维卷积神经网络(3DCNN)对微表情序列进行处理,获取兼顾时间域和空间域的鲁棒性特征。构建多尺度时间输入序列,从不同时间长度的图像序列中提取多维时间特征,采用轻量级3DCNN提取多尺度时空特征,利用全局时空注意力模块(GSAM)对时空特征进行全局性时空关联加强,其中时空重组模块用于加强不同时刻图像帧之间的连通性,全局信息关注模块构建单帧图像上的空间关联信息,最后对不同时刻的特征赋予权重以突出关键时间信息,有效完成微表情检测工作。实验结果表明,该方法可以准确检测出微表情序列片段,在CASME、CASME II和SAMM公开数据集上的准确率分别达到92.32%、95.04%和89.56%,相比目前最优的深度学习方法LGAttNet,所提方法在CASME II和SAMM数据集上的准确率分别提高了3.84和4.96个百分点。

关键词: 微表情检测, 三维卷积神经网络, 时空特征, 多尺度特征, 关联性

Abstract: Micro-expressions can reveal genuine emotions that people attempt to hide, providing potential information for criminal investigations, psychological counseling, and other situations. Existing methods for detecting micro-expression primarily extract temporal characteristics to construct spatiotemporal features based on obtaining spatial features; however, these approaches can result in distorted temporal features, and thus disrupt the original temporal relationships during spatial processing, consequently diminishing the discriminative ability of the spatiotemporal features of micro-expressions. To address this issue, a method is proposed for micro-expression detection based on a multi-scale spatiotemporal attention network. Using a 3-Dimensional Convolutional Neural Network (3DCNN) that incorporates temporal and spatial relationships, the micro-expression sequences are processed to obtain robust features considering both the temporal and spatial domains. Multi-scale temporal input sequences are constructed to extract multi-dimensional temporal features from image sequences with different time lengths in the network. A lightweight 3DCNN is used to extract multi-scale spatiotemporal features. The Global Spatiotemporal Attention Module (GSAM) is employed to enhance the overall spatiotemporal correlations of features, wherein the spatiotemporal restructuring module strengthens the connectivity between different image frames at different moments, whereas the global information attention module constructs the spatial correlation information on a single-frame image. Finally, the assignment of weights to various temporal characteristics highlights the key temporal information, effectively detecting micro-expressions. The experimental results demonstrate that the proposed method can accurately detect micro-expression sequence fragments, achieving accuracy rates of 92.32%, 95.04%, and 89.56% on the publicly available CASME, CASME II, and SAMM datasets, respectively. Compared with that of the existing optimal deep learning method, LGAttNet, the accuracy of the proposed method is improved by 3.84 percentage points on the CASME II dataset and 4.96 percentage points on the SAMM dataset.

Key words: micro-expression detection, 3-Dimensional Convolutional Neural Network(3DCNN), spatiotemporal features, multi-scale features, correlation

中图分类号:

TP391

于洋, 孙芳芳, 吕华, 李扬, 王晓民. 基于多尺度时空注意力网络的微表情检测方法[J]. 计算机工程, 2024, 50(6): 228-235.

YU Yang, SUN Fangfang, LV Hua, LI Yang, WANG Xiaomin. Micro-Expression Detection Method Based on Multi-Scale Spatiotemporal Attention Network[J]. Computer Engineering, 2024, 50(6): 228-235.

https://www.ecice06.com/CN/Y2024/V50/I6/228

参考文献

[1] GONG W J, AN Z H, ELFIKY N M. Deep learning-based microexpression recognition:a survey[J]. Neural Computing and Applications, 2022, 34(12):9537-9560.
[2] 王甦菁,邹博超,刘瑞,等.隐藏情绪分析与识别方法[J].心理科学进展, 2020, 28(9):1426-1436. WANG S J, ZOU B C, LIU R, et al. Concealed emotion analysis and recognition method[J]. Advances in Psychological Science, 2020, 28(9):1426-1436.(in Chinese)
[3] ZHOU L, SHAO X Y, MAO Q R. A survey of micro-expression recognition[J]. Image and Vision Computing, 2021, 105:104043.
[4] LIONG S T, SEE J, WONG K, et al. Automatic apex frame spotting in micro-expression database[C]//Proceedings of the 3rd IAPR Asian Conference on Pattern Recognition. Washington D.C.,USA:IEEE Press,2015:665-669.
[5] LIU Y J, ZHANG J K, YAN W J, et al. A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2016, 7(4):299-310.
[6] LIU Y J, LI B J, LAI Y K. Sparse MDMO:learning a discriminative feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing, 2021,12(1):254-261.
[7] LI J T, SOLADIE C, SEGUIER R. LTP-ML:micro-expression detection by recognition of local temporal pattern of facial movements[C]//Proceedings of the 13th IEEE International Conference on Automatic Face&Gesture Recognition. Washington D.C.,USA:IEEE Press,2018:634-641.
[8] WEI J S, LU G M, YAN J J, et al. Micro-expression recognition using local binary pattern from five intersecting planes[J]. Multimedia Tools and Applications, 2022, 81(15):20643-20668.
[9] CEN S X, YU Y, YAN G, et al. Multi-task facial activity patterns learning for micro-expression recognition using joint temporal local cube binary pattern[J]. Signal Processing:Image Communication, 2022, 103:116616.
[10] DAVISON A, MERGHANI W, LANSLEY C, et al. Objective micro-facial movement detection using FACS-based regions and baseline evaluation[C]//Proceedings of the 13th IEEE International Conference on Automatic Face&Gesture Recognition. Washington D.C.,USA:IEEE Press,2018:642-649.
[11] DAVISON A, MERGHANI W, YAP M. Objective classes for micro-facial expression recognition[J]. Journal of Imaging, 2018, 4(10):119.
[12] HUANG X H, ZHAO G Y, HONG X P, et al. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[J]. Neurocomputing, 2016, 175:564-578.
[13] LU H, LI D W, KPALMA K, et al. Geometric distance for fast micro-expression detection[C]//Proceedings of the 20th IEEE International Conference on Communication Technology. Washington D.C.,USA:IEEE Press,2020:1405-1411.
[14] PATEL D, ZHAO G Y, PIETIKÄINEN M. Spatiotemporal integration of optical flow vectors for micro-expression detection[EB/OL].[2023-06-05].https://link.springer.com/chapter/10.1007/978-3-319-25903-1_32.
[15] 于明,钟元想,王岩.人脸微表情分析方法综述[J].计算机工程, 2023, 49(2):1-14. YU M, ZHONG Y X, WANG Y. A survey of facial micro-expression analysis methods[J]. Computer Engineering, 2023, 49(2):1-14.(in Chinese)
[16] BORZA D, ITU R, DANESCU R. Micro expression detection and recognition from high speed cameras using convolutional neural networks[C]//Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications. Washington D.C.,USA:IEEE Press,2018:201-208.
[17] SUN H, LI B, CHEN L, et al. Micro-expression detection based on action units and multi-region feature fusion[C]//Proceedings of the 5th International Conference on Image and Graphics Processing. Washington D.C.,USA:IEEE Press,2022:30-36.
[18] VERBURG M, MENKOVSKI V. Micro-expression detection in long videos using optical flow and recurrent neural networks[C]//Proceedings of the 14th IEEE International Conference on Automatic Face&Gesture Recognition. Washington D.C.,USA:IEEE Press,2019:1-6.
[19] ZHOU Y, SONG Y X, CHEN L, et al. A novel micro-expression detection algorithm based on BERT and 3DCNN[J]. Image and Vision Computing, 2022, 119:104378.
[20] WANG S J, HE Y, LI J T, et al. MESNet:a convolutional neural network for spotting multi-scale micro-expression intervals in long videos[J]. IEEE Transactions on Image Processing, 2021, 30:3956-3969.
[21] TAKALKAR M A, THUSEETHAN S, RAJASEGARAR S, et al. LGAttNet:automatic micro-expression detection using dual-stream local and global attentions[J]. Knowledge-Based Systems, 2021, 212:106566.
[22] YAN W J, WU Q, LIU Y J, et al. CASME database:a dataset of spontaneous micro-expressions collected from neutralized faces[C]//Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Washington D.C.,USA:IEEE Press,2013:1-7.
[23] YAN W J, LI X B, WANG S J, et al. CASME II:an improved spontaneous micro-expression database and the baseline evaluation[J]. PLoS One, 2014, 9(1):e86041.
[24] DAVISON A K, LANSLEY C, COSTEN N, et al. SAMM:a spontaneous micro-facial movement dataset[J]. IEEE Transactions on Affective Computing, 2018, 9(1):116-129.
[25] OJALA T, PIETIKÄINEN M, MÄENPÄÄ T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(7):971-987.
[26] LU H, KPALMA K, RONSIN J. Micro-expression detection using integral projections[J]. Journal of WSCG, 2017, 25(2):87-96.
[27] ZHANG Z H, CHEN T, MENG H Y, et al. SMEConvNet:a convolutional neural network for spotting spontaneous facial micro-expression from long videos[J]. IEEE Access, 2018, 6:71143-71151.
[28] LI X H, YU J, ZHAN S. Spontaneous facial micro-expression detection based on deep learning[C]//Proceedings of the 13th IEEE International Conference on Signal Processing. Washington D.C.,USA:IEEE Press,2016:1130-1134.

选择文件类型/文献管理软件名称

选择包含的内容