Research on Reconstruction Method of Occluded Micro-Expressions Based on Quantized Denoising Autoencoder

doi:10.19678/j.issn.1000-3428.0068949

Abstract

Abstract:

Micro-expressions serve as a crucial basis for psychological health diagnoses, and occlusions caused by objects, such as glasses or masks, can make micro-expression recognition challenging. Existing reconstruction methods for occluded micro-expressions rely primarily on reconstructing RGB texture information, which leads to issues such as information redundancy and difficulties in achieving precise texture reconstruction. In addition, the models used in such reconstruction methods often involve symmetric autoencoders based on U-Net and Generative Adversarial Networks (GAN); However, the former suffers from limited reconstruction capabilities in shallow symmetric structures, and the latter faces challenges in terms of adversarial loss convergence speed. This paper proposes a method for reconstructing dynamic flow features in occluded regions of micro-expressions based on a vector-quantized denoising autoencoder. First, dynamic flow, a robust feature representation resilient to lighting variations, is proposed based on optical flow and dynamic images, effectively aggregating motion information from all TVL1 optical flows and simplifying the texture information. Then, a two-pair Vector Quantized Denoising Autoencoder (VQ-DAE) based on the discrete encoding Vector Quantized Variational Autoencoder (VQ-VAE) is introduced to reconstruct dynamic flow features in occluded regions of micro-expressions to facilitate the recognition of occluded micro-expressions. Experimental results demonstrate that this approach effectively reconstructs motion information in occluded regions, achieving accuracy rates of 77.89%, 72.02%, and 61.04% on the CASME, CAS(ME)², and CASME Ⅱ datasets, respectively. Compared to traditional, spatial-attention-, and self-attention-based methods, our method leads to significant improvements in accuracy, Unweighted Average Recall (UAR), and Macro-F1.

Key words: occluded micro-expression recognition, feature reconstruction, optical flow, dynamic image, denoising autoencoder

摘要：

微表情是一种心理健康诊断的重要依据, 眼镜、口罩等物体造成的遮挡会导致微表情识别困难。现有遮挡微表情重建方法以RGB纹理信息重建为主, 存在信息大量冗余、难以实现对纹理的精确重建等问题。此外, 重建方法采用的模型多为基于U-Net的对称自编码器和生成对抗网络(GAN)等, 存在浅层的对称结构重建能力有限、对抗损失收敛困难等问题。为此, 提出一种基于量子化降噪自编码器的微表情遮挡区域动态流特征重建方法。首先, 基于光流和动态图像提出光照能量鲁棒的动态流特征表示, 有效聚合所有TVL1光流中的运动信息, 并简化纹理信息; 其次, 基于离散编码的变分自编码器(VQ-VAE)提出一种双层结构向量量子化降噪自编码器(VQ-DAE), 用于微表情的遮挡区域动态流特征重建, 以进行遮挡微表情的识别。实验结果表明, 该方法能较好地重建遮挡区域的运动信息, 在CASME、CAS(ME)²、CASME Ⅱ这3个数据集上的准确率分别达到77.89%、72.02%、61.04%。与传统方法、基于空间注意力及自注意力方法相比, 所提方法在准确率、未加权平均召回率(UAR)、Macro-F1等指标上均有显著的性能提升。

关键词: 遮挡微表情识别, 特征重建, 光流, 动态图像, 降噪自编码器

LIU Hui, GUO Te, LIU Dong, LI Yingying. Research on Reconstruction Method of Occluded Micro-Expressions Based on Quantized Denoising Autoencoder[J]. Computer Engineering, 2025, 51(5): 288-304.

刘慧, 郭特, 刘栋, 李颖颖. 基于量子化降噪自编码器的遮挡微表情重建方法研究[J]. 计算机工程, 2025, 51(5): 288-304.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0068949

https://www.ecice06.com/EN/Y2025/V51/I5/288

Figures/Tables 18

Fig.1 Dynamic flow feature reconstruction framework of micro-expression based on vector quantization denoising autoencoder

Fig.2 Examples of micro-expression sample occlusion in CASME dataset

Fig.3 Farneback & TVL1 optical flows of a CASME Ⅱ sample

Fig.4 Dynamic flow feature visualization of partial micro-expression samples

Fig.5 Optical flow segmentation for extended dynamic flow

Fig.6 Overall architecture of VQ-DAE

Fig.7 Encoder 1 and Decoder 2 at the bottom level

Fig.8 Decoder 1 and Encoder 2 at the top level and Upscaler, CodeBook, ReZero unit

Fig.9 Comparison of Farneback optical flow reconstruction of UNet-DAE and dynamic flow reconstruction of VQ-DAE

Fig.10 Confusion matrix of recognition accuracy

Fig.11 Comparison of convergence rates of different residual structures

Fig.12 Comparison of reconstruction results with different levels

Fig.13 Comparisons of reconstruction results with different encoding methods

References 60

1	LIES T. Clues to deceit in the marketplace, politics, and marriage[M]. New York, USA: [s. n. ], 1992.
2	EKMAN P , FRIESEN W V . Constants across cultures in the face and emotion. PLoS One, 1971, 17 (2): 124- 129. doi: 10.1037/h0030377
3	FERRETTI V , PAPALEO F . Understanding others: emotion recognition in humans and other animals. Genes, Brain and Behavior, 2019, 18 (1): e12544. doi: 10.1111/gbb.12544
4	GOTTMAN J M , LEVENSON R W . A two-factor model for predicting when a couple will divorce: exploratory analyses using 14-year longitudinal data. Family Process, 2002, 41 (1): 83- 96. doi: 10.1111/j.1545-5300.2002.40102000083.x
5	SALTER F , GRAMMER K , RIKOWSKI A . Sex differences in negotiating with powerful males: an ethological analysis of approaches to nightclub doormen. Human Nature (Hawthorne, N Y), 2005, 16 (3): 306- 321. doi: 10.1007/s12110-005-1013-4
6	陈庄, 赵源, 罗颂, 等. 双通道动静态特征的微表情识别. 小型微型计算机系统, 2023, 44 (7): 1500- 1507.
	CHEN Z , ZHAO Y , LUO S , et al. Micro-expression recognition based on dynamic and static features of two channels. Journal of Chinese Computer Systems, 2023, 44 (7): 1500- 1507.
7	PORTER S , TEN BRINKE L . Reading between the lies: identifying concealed and falsified emotions in universal facial expressions. Psychological Science, 2008, 19 (5): 508- 514. doi: 10.1111/j.1467-9280.2008.02116.x
8	LU Y , WANG S G , ZHAO W T , et al. WGAN-based robust occluded facial expression recognition. IEEE Access, 2019, 7, 93594- 93610. doi: 10.1109/ACCESS.2019.2928125
9	CORNEJO J Y R, PEDRINI H. Emotion recognition based on occluded facial expressions[EB/OL]. [2023-09-05]. https://link.springer.com/chapter/10.1007/978-3-319-68560-1_28.
10	POUX D , ALLAERT B , IHADDADENE N , et al. Dynamic facial expression recognition under partial occlusion with optical flow reconstruction. IEEE Transactions on Image Processing, 2022, 31, 446- 457. doi: 10.1109/TIP.2021.3129120
11	MATHIS A, NOTHWANG W, DONAVANIK D, et al. Making optic flow robust to dynamic lighting conditions for real-time operation[EB/OL]. [2023-09-05]. https://apps.dtic.mil/sti/tr/pdf/AD1005369.pdf.
12	SAXENA D , CAO J N . Generative Adversarial Networks (GANs): challenges, solutions, and future directions. ACM Computing Surveys, 2021, 54 (3): 1- 42.
13	LIU S S, ZHANG Y, LIU K P, et al. Facial expression recognition under partial occlusion based on Gabor multi-orientation features fusion and local Gabor binary pattern histogram sequence[C]//Proceedings of the 9th International Conference on Intelligent Information Hiding and Multimedia Signal Processing. Washington D.C., USA: IEEE Press, 2013: 218-222.
14	ZHANG L G , TJONDRONEGORO D , CHANDRAN V . Random Gabor based templates for facial expression recognition in images with facial occlusion. Neurocomputing, 2014, 145, 451- 464. doi: 10.1016/j.neucom.2014.05.008
15	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL]. [2023-09-05]. https://arxiv.org/abs/1706.03762.
16	LI Y, ZENG J B, SHAN S G, et al. Patch-gated CNN for occlusion-aware facial expression recognition[C]//Proceedings of the 24th International Conference on Pattern Recognition. Washington D.C., USA: IEEE Press, 2018: 2209-2214.
17	DING H, ZHOU P, CHELLAPPA R, et al. Occlusion-adaptive deep network for robust facial expression recognition[C]//Proceedings of the 2020 IEEE International Joint Conference on Biometrics. New York, USA: ACM Press, 2020: 1-9.
18	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[EB/OL]. [2023-09-05]. https://arxiv.org/abs/2010.11929.
19	FARZANEH A H, QI X J. Facial expression recognition in the wild via deep attentive center loss[C]//Proceedings of the IEEE Winter Conference on Applications of Computer Vision. Washington D.C., USA: IEEE Press, 2021: 2402-2411.
20	李晶, 李健, 陈海丰, 等. 基于关键区域遮挡与重建的人脸表情识别. 计算机工程, 2024, 50 (5): 241- 249. doi: 10.19678/j.issn.1000-3428.0067538
	LI J , LI J , CHEN H F , et al. Facial expression recognition based on key region masking and reconstruction. Computer Engineering, 2024, 50 (5): 241- 249. doi: 10.19678/j.issn.1000-3428.0067538
21	MA F Y , SUN B , LI S T . Facial expression recognition with visual transformers and attentional selective fusion. IEEE Transactions on Affective Computing, 2023, 14 (2): 1236- 1248. doi: 10.1109/TAFFC.2021.3122146
22	GAO J , ZHAO Y . TFE: a transformer architecture for occlusion aware facial expression recognition. Frontiers in Neurorobotics, 2021, 15, 763100. doi: 10.3389/fnbot.2021.763100
23	LIU C , HIROTA K , DAI Y P . Patch attention convolutional vision transformer for facial expression recognition with occlusion. Information Sciences, 2023, 619, 781- 794. doi: 10.1016/j.ins.2022.11.068
24	LI H T, SUI M Z, ZHAO F, et al. MVT: mask vision transformer for facial expression recognition in the wild[EB/OL]. [2023-09-05]. https://arxiv.org/abs/2106.04520v2.
25	ZHENG C, MENDIETA M, CHEN C. POSTER: a pyramid cross-fusion transformer network for facial expression recognition[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2023: 3146-3155.
26	ZHANG L F , HONG X P , ARANDJELOVIC O , et al. Short and long range relation based spatio-temporal transformer for micro-expression recognition. IEEE Transactions on Affective Computing, 2022, 13 (4): 1973- 1985. doi: 10.1109/TAFFC.2022.3213509
27	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2018: 7132-7141.
28	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[EB/OL]. [2023-09-05]. https://arxiv.org/abs/1807.06521.
29	ZHAO Z Q , LIU Q S , WANG S M . Learning deep global multi-scale and local attention features for facial expression recognition in the wild. IEEE Transactions on Image Processing, 2021, 30, 6544- 6556. doi: 10.1109/TIP.2021.3093397
30	GERA D , BALASUBRAMANIAN S . Landmark guidance independent spatio-channel attention and complementary context information based facial expression recognition. Pattern Recognition Letters, 2021, 145, 58- 66. doi: 10.1016/j.patrec.2021.01.029
31	WEN Z , LIN W , WANG T , et al. Distract your attention: multi-head cross attention network for facial expression recognition. Biomimetics (Basel, Switzerland), 2023, 8 (2): 199. doi: 10.3390/biomimetics8020199
32	TOWNER H, SLATER M. Reconstruction and recognition of occluded facial expressions using PCA[EB/OL]. [2023-09-05]. https://link.springer.com/chapter/10.1007/978-3-540-74889-2_4.
33	LIN J C, WU C H, WEI W L. Facial action unit prediction under partial occlusion based on error weighted cross-correlation model[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D.C., USA: IEEE Press, 2013: 3482-3486.
34	CORNEJO J Y R, PEDRINI H. Emotion recognition based on occluded facial expressions[EB/OL]. [2023-09-05]. https://link.springer.com/chapter/10.1007/978-3-319-68560-1_28.
35	MAO X, XUE Y L, LI Z, et al. Robust facial expression recognition based on RPCA and AdaBoost[C]//Proceedings of the 10th Workshop on Image Analysis for Multimedia Interactive Services. Washington D.C., USA: IEEE Press, 2009: 113-116.
36	JIANG B, JIA K B. Research of robust facial expression recognition under facial occlusion condition[EB/OL]. [2023-09-05]. https://link.springer.com/chapter/10.1007/978-3-642-23620-4_13.
37	JIANG M Y, WANG Y W, MCKEOWN M J, et al. Occlusion-robust FAU recognition by mining latent space of masked autoencoders[EB/OL]. [2023-09-05]. https://arxiv.org/abs/2212.04029v1.
38	杨鲁月, 张树美, 赵俊莉. 基于并行Gan的有遮挡动态表情识别. 计算机工程与应用, 2021, 57 (24): 168- 178.
	YANG L Y , ZHANG S M , ZHAO J L . Dynamic expression recognition with partial occlusion based on parallel Gan. Computer Engineering and Applications, 2021, 57 (24): 168- 178.
39	MA B W, AN R D, ZHANG W, et al. Facial action unit detection and intensity estimation from self-supervised representation[EB/OL]. [2023-09-05]. https://arxiv.org/abs/2210.15878v1.
40	王海涌, 梁红珠. 基于改进的GAN的局部遮挡人脸表情识别. 计算机工程与应用, 2020, 56 (5): 141- 146.
	WANG H Y , LIANG H Z . Recognition of local occluded facial expressions based on improved generative adversarial network. Computer Engineering and Applications, 2020, 56 (5): 141- 146.
41	YAN W J, WU Q, LIU Y J, et al. CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces[C]//Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Washington D.C., USA: IEEE Press, 2013: 1-7.
42	QU F B , WANG S J , YAN W J , et al. CAS(ME)²: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Transactions on Affective Computing, 2017, 9 (4): 424- 436.
43	LI X B, PFISTER T, HUANG X H, et al. A spontaneous micro-expression database: inducement, collection and baseline[C]//Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Washington D.C., USA: IEEE Press, 2013: 1-6.
44	FARNEBÄCK G. Two-frame motion estimation based on polynomial expansion[EB/OL]. [2023-09-05]. https://link.springer.com/chapter/10.1007/3-540-45103-X_50.
45	ZACH C, POCK T, BISCHOF H. A duality based approach for realtime TV-L 1 optical flow[EB/OL]. [2023-09-05]. https://link.springer.com/chapter/10.1007/978-3-540-74936-3_22.
46	BILEN H, FERNANDO B, GAVVES E, et al. Dynamic image networks for action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2016: 3034-3042.
47	GARBACEA C, VAN DEN OORD A, LI Y Z, et al. Low bit-rate speech coding with VQ-VAE and a WaveNet decoder[C]//Proceedings of the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D.C., USA: IEEE Press, 2019: 735-739.
48	CHEN Y J, CHENG SHIN-I, CHIU W C, et al. Vector quantized image-to-image translation[EB/OL]. [2023-09-05]. https://arxiv.org/abs/2207.13286.
49	VAN DEN OORD A, VINYALS O, KAVUKCUOGLU K, et al. Neural discrete representation learning[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 6309-6318.
50	VAN DEN OORD A, KALCHBRENNER N, KAVUKCUOGLU K, et al. Pixel recurrent neural networks[C]//Proceedings of the 33rd International Conference on Machine Learning. New York, USA: ACM Press, 2016: 1747-1756.
51	RAZAVI A, VAN DEN OORD A, VINYALS O. Generating diverse highfidelity images with VQ-VAE-2[EB/OL]. [2023-09-05]. https://arxiv.org/abs/1906.00446.
52	SHANNON C. Coding theorems for a discrete source with a fidelity criterion[EB/OL]. [2023-09-05]. https://gwern.net/doc/cs/algorithm/information/1959-shannon.pdf.
53	BACHLECHNER T, MAJUMDER B P, MAO H H, et al. ReZero is all you need: fast convergence at large depth[EB/OL]. [2023-09-05]. https://arxiv.org/abs/2003.04887v2.
54	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2016: 770-778.
55	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2023-09-05]. https://arxiv.org/abs/1409.1556v6.
56	SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2016: 2818-2826.
57	ZHAO G , PIETIKÄINEN M . Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29 (6): 915- 928.
58	CHAUDHRY R, RAVICHANDRAN A, HAGER G, et al. Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2009: 1932-1939.
59	LIONG S T , SEE J , WONG K , et al. Less is more: micro-expression recognition from video using Apex frame. Signal Processing: Image Communication, 2018, 62, 82- 92. doi: 10.1016/j.image.2017.11.006
60	LIU Y J , ZHANG J K , YAN W J , et al. A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Transactions on Affective Computing, 2016, 7 (4): 299- 310.

[1]	Lulu YANG, Ping MA, Cong WANG, Xinkai LI, Yue MENG, Hongli ZHANG. Insulator Detection Method Using YOLO Combining Feature Reuse and Reconstruction [J]. Computer Engineering, 2024, 50(7): 303-313.
[2]	YANG Shuo, WANG Yiding. Facial Animation Algorithm Based on Improved Thin Plate Spline Motion Model [J]. Computer Engineering, 2024, 50(6): 255-265.
[3]	ZHOU Qinyuan, DENG Yueping, ZHANG Lei, ZHANG Chen, LU Rirong, HU Xianzhe. Dynamic Visual SLAM System Integrating Optical Flow and Multi-View Geometry [J]. Computer Engineering, 2024, 50(5): 250-259.
[4]	DU Tiantian, WANG Xiaolong, HE Jing. Optical-flow-based Waterway Velocity Detection Algorithm Under Complex Illumination Conditions [J]. Computer Engineering, 2024, 50(4): 60-67.
[5]	Chunbo XU, Juan YAN, Huibin YANG, Bo WANG, Han WU. Visual SLAM Algorithm Based on Target Detection and Semantic Segmentation [J]. Computer Engineering, 2023, 49(8): 199-206, 214.
[6]	WU Zijia, CHEN Hang, PENG Yong, SONG Wei. Visual SLAM with Lightweight YOLOv5s in Dynamic Environment [J]. Computer Engineering, 2022, 48(8): 187-195,205.
[7]	LI Jie, ZHOU Shun. Orthophoto Splicing Method Guided by Superpixel and Optical Flow [J]. Computer Engineering, 2022, 48(3): 263-270.
[8]	LIANG Zhengyou, LIU Dezhi, SUN Yu. Micro-Expression Recognition Method Combining Transfer Learning and Separable 3D Convolution [J]. Computer Engineering, 2022, 48(1): 228-235.
[9]	JI Haoxuanye, LIANG Pengpeng, CHAI Yumei, WANG Liming. Planar Object Tracking Algorithm Based on Key Points and Optical Flow [J]. Computer Engineering, 2021, 47(4): 234-240.
[10]	LUO Fanbo, WANG Ping, LIANG Siyuan, XU Guifei, WANG Wei. Crowd Abnormal Behavior Recognition Based on Deep Learning and Sparse Optical Flow [J]. Computer Engineering, 2020, 46(4): 287-293,300.
[11]	HU Fei,WEN Chang,XIE Kai,HE Jianbiao. Multi-cue fusion face liveness detection based on fine-tuning strategy [J]. Computer Engineering, 2019, 45(5): 256-260.
[12]	JIN Zefenfen,HOU Zhiqiang,YU Wangsheng,WANG Xin. Color and Motion Features Fusion Tracking Based on Game Theory [J]. Computer Engineering, 2018, 44(8): 237-242.
[13]	WANG Shuyuana,TIAN Shengwei,YU Long,FENG Guanjun,AISHAN Wumaier,LI Pu,ZHAO Jianguo. Identification of Uyghur Event Coreference Relationship Based on Stacked Denoising Autoencoder [J]. Computer Engineering, 2018, 44(6): 305-310.
[14]	XU Weiwei,LI Jun. A Robust Real-time Tracking Method of Facial Key Point [J]. Computer Engineering, 2018, 44(4): 281-286.
[15]	ZHANG Wenda,XU Yuelei,MA Shiping,LI Shuai,ZOU Hongzhong. Optical Flow Calculating Method Based on Multi-scale V1-MT Feedforward Model [J]. Computer Engineering, 2017, 43(9): 205-209.

Please choose a citation manager

Content to export