基于关键区域遮挡与重建的人脸表情识别

doi:10.19678/j.issn.1000-3428.0067538

计算机工程 ›› 2024, Vol. 50 ›› Issue (5): 241-249. doi: 10.19678/j.issn.1000-3428.0067538

基于关键区域遮挡与重建的人脸表情识别

李晶¹, 李健¹, 陈海丰¹, 张倩¹, 王丽燕², 裴二成³

1. 陕西科技大学电子信息与人工智能学院, 陕西西安 710021;
2. 陕西科技大学文理学院, 陕西西安 710021;
3. 西安邮电大学计算机学院, 陕西西安 710100

收稿日期:2023-05-04 修回日期:2023-08-25 发布日期:2023-10-10
通讯作者: 李晶,E-mail:211612105@sust.edu.cn E-mail:211612105@sust.edu.cn
基金资助:
国家自然科学基金(62306172);国家土建结构预制装配化工程技术研究中心沈祖炎专项基金(2019CPCCE-K02);陕西省自然科学基础研究计划项目(2022JQ-662);2021年陕西科技大学教育信息化教学改革研究项目(JXJG2021-09);陕西科技大学博士科研启动基金(126022325)。

Facial Expression Recognition Based on Key Region Masking and Reconstruction

LI Jing¹, LI Jian¹, CHEN Haifeng¹, ZHANG Qian¹, WANG Liyan², PEI Ercheng³

1. School of Electronic Information and Artificial Intelligence, Shaanxi University of Science and Technology, Xi'an 710021, Shaanxi, China;
2. School of Arts and Sciences, Shaanxi University of Science and Technology, Xi'an 710021, Shaanxi, China;
3. School of Computer Science and Technology, Xi'an University of Posts and Telecommunications, Xi'an 710100, Shaanxi, China

Received:2023-05-04 Revised:2023-08-25 Published:2023-10-10
Contact: 李晶,E-mail:211612105@sust.edu.cn E-mail:211612105@sust.edu.cn

摘要/Abstract

摘要： 为了解决自然场景下人脸表情识别任务中的无用信息干扰和遮挡对识别性能的影响问题,提出一种基于关键区域遮挡与重建的人脸表情识别模型。利用多尺度特征提取网络,提取人脸图像的全局特征。根据68个人脸关键点划分出68个关键区域,并通过插值法提取68个关键区域的特征,同时采用注意力机制学习关键区域特征之间的先验关系。设计自监督的遮挡与重建模块,对关键区域特征进行随机遮挡,并利用已知区域信息来预测和重建被遮挡区域的特征,从而提高模型在自然场景下的表情识别性能。设计多个实验验证了该模型的泛化能力,并通过消融实验验证了模型中每个模块的有效性。实验结果表明,该模型在真实世界的情感面孔数据集(RAF-DB)和Occlusion-RAF-DB数据集上分别达到了88.44%和86.09%的识别准确率,相比于视觉Transformer(ViT)等模型有效地提升了自然场景下人脸表情识别的性能。

关键词: 人脸表情识别, 多尺度关键区域特征, 注意力机制, 自监督学习, 遮挡与重建

Abstract: To overcome the negative impact of irrelevant information interference and masking issues on the performance of facial expression recognition in the wild, this study proposes a facial expression recognition model based on key region masking and reconstruction. A multi-scale feature extraction network is first used to extract global features from facial images. Thereafter, the features of key regions, based on 68 facial landmarks, are extracted and encoded with attention mechanisms to learn prior relationships between the features of the key regions. To further enhance the discriminative capability of the extracted features for improved recognition performance, a key region masking and reconstruction module is designed based on self-supervised learning. This module aims to reconstruct randomly masked features of key regions using known region information. Extensive experiments are conducted to validate the generalization ability of the model, and ablation experiments confirm the effectiveness of each module in the model. The experimental results demonstrate that the model achieves recognition accuracies of 88.44% and 86.09% on the Real-world Affective Faces DataBase(RAF-DB) and the Occlusion-RAF-DB dataset, respectively, effectively improving the performance of facial expression recognition in natural scenarios compared to models such as Vision Transformer(ViT).

Key words: facial expression recognition, multiscale key region feature, attention mechanism, self-supervised learning, masking and reconstruction

中图分类号:

TP391.41

李晶, 李健, 陈海丰, 张倩, 王丽燕, 裴二成. 基于关键区域遮挡与重建的人脸表情识别[J]. 计算机工程, 2024, 50(5): 241-249.

LI Jing, LI Jian, CHEN Haifeng, ZHANG Qian, WANG Liyan, PEI Ercheng. Facial Expression Recognition Based on Key Region Masking and Reconstruction[J]. Computer Engineering, 2024, 50(5): 241-249.

https://www.ecice06.com/CN/Y2024/V50/I5/241

参考文献

[1] MELLOUK W, HANDOUZI W. Facial emotion recognition using deep learning:review and insights[J]. Procedia Computer Science, 2020, 175:689-694.
[2] TANG X Y, PENG W Y, LIU S R, et al. Classroom teaching evaluation based on facial expression recognition[C]//Proceedings of the 9th International Conference on Educational and Information Technology. New York, USA:ACM Press, 2020:62-67.
[3] AL-EIDAN R M, AL-KHALIFA H, AL-SALMAN A. Deep-learning-based models for pain recognition:a systematic review[J]. Applied Sciences, 2020, 10(17):5984.
[4] HAPPY S L, ROUTRAY A. Robust facial expression classification using shape and appearance features[C]//Proceedings of the 8th International Conference on Advances in Pattern Recognition. Washington D.C., USA:IEEE Press, 2015:1-5.
[5] 罗思诗, 李茂军, 陈满. 多尺度融合注意力机制的人脸表情识别网络[J]. 计算机工程与应用, 2023, 59(1):199-206. LUO S S, LI M J, CHEN M. Multi-scale integrated attention mechanism for facial expression recognition network[J]. Computer Engineering and Applications, 2023, 59(1):199-206.(in Chinese)
[6] CAKIR D, YILMAZ G, ARICA N. Facial action unit detection with ViT and perceiver using landmark patches[C]//Proceedings of the 12th Annual Information Technology, Electronics and Mobile Communication Conference. Washington D.C., USA:IEEE Press, 2021:281-285.
[7] CHEN H F, JIANG D M, ZHAO Y, et al. Region attentive action unit intensity estimation with uncertainty weighted multi-task learning[J]. IEEE Transactions on Affective Computing, 2023, 14(3):2033-2047.
[8] ZHU X L, HE Z L, ZHAO L, et al. A cascade attention based facial expression recognition network by fusing multi-scale spatio-temporal features[J]. Sensors, 2022, 22(4):1350.
[9] CHEN Y J, LIU S G. Deep partial occlusion facial expression recognition via improved CNN[M]. Berlin, Germany:Springer, 2020.
[10] SONG L X, GONG D H, LI Z F, et al. Occlusion robust face recognition based on mask learning with pairwise differential Siamese network[EB/OL].[2023-04-11]. https://arxiv.org/pdf/1908.06290.pdf.
[11] HE K M, CHEN X L, XIE S N, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA:IEEE Press, 2022:16000-16009.
[12] ZHAO K L, CHU W S, ZHANG H G. Deep region and multi-label learning for facial action unit detection[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA:IEEE Press, 2016:3391-3399.
[13] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA:ACM Press, 2017:6000-6010.
[14] ZAVASCHI T H H, KOERICH A L, OLIVEIRA L E S. Facial expression recognition using ensemble of classifiers[C]//Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D.C., USA:IEEE Press, 2011:1489-1492.
[15] CHEN S K, WANG J F, CHEN Y D, et al. Label distribution learning on auxiliary label space graphs for facial expression recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA:IEEE Press, 2020:13984-13993.
[16] ZHAO Z, LIU Q, ZHOU F. Robust lightweight facial expression recognition network with label distribution training[C]//Proceedings of AAAI Conference on Artificial Intelligence. Palo Alto, USA:AAAI Press, 2021:3510-3519.
[17] WANG K, PENG X, YANG J, et al. Region attention networks for pose and occlusion robust facial expression recognition[J]. IEEE Transactions on Image Processing, 2020, 29:4057-4069.
[18] LIANG X, XU L, LIU J, et al. Patch attention layer of embedding handcrafted features in CNN for facial expression recognition[J]. Sensors, 2021, 21(3):833.
[19] YANG W, GAO H W, JIANG Y Q, et al. A cascaded feature pyramid network with non-backward propagation for facial expression recognition[J]. IEEE Sensors Journal, 2021, 21(10):11382-11392.
[20] DING H, ZHOU P, CHELLAPPA R. Occlusion-adaptive deep network for robust facial expression recognition[C]//Proceedings of 2020 IEEE International Joint Conference on Biometrics. New York, USA:ACM Press, 2020:1-9.
[21] 王军, 赵凯, 程勇. 基于遮挡感知卷积神经网络的面部表情识别模型[J]. 计算机工程, 2021, 47(10):242-251. WANG J, ZHAO K, CHENG Y. Facial expression recognition model based on convolutional neural network with occlusion perception[J]. Computer Engineering, 2021, 47(10):242-251.(in Chinese)
[22] KIM J, LEE D. Facial expression recognition robust to occlusion and to intra-similarity problem using relevant subsampling[J]. Sensors, 2023, 23(5):2619.
[23] CHEN Y A, CHEN W C, WEI C P, et al. Occlusion-aware face inpainting via generative adversarial networks[C]//Proceedings of IEEE International Conference on Image Processing. Washington D.C., USA:IEEE Press, 2017:1202-1206.
[24] LU Y, WANG S G, ZHAO W T, et al. WGAN-based robust occluded facial expression recognition[J]. IEEE Access, 2019, 7:93594-93610.
[25] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words:Transformers for image recognition at scale[EB/OL].[2023-04-11]. https://arxiv.org/abs/2010.11929.
[26] LI S, DENG W H, DU J P. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA:IEEE Press, 2017:2852-2861.
[27] WANG K, PENG X J, YANG J F, et al. Suppressing uncertainties for large-scale facial expression recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA:IEEE Press, 2020:6897-6906.
[28] MA F Y, SUN B, LI S T. Facial expression recognition with visual transformers and attentional selective fusion[J]. IEEE Transactions on Affective Computing, 2023, 14(2):1236-1248.
[29] 冉瑞生, 翁稳稳, 王宁, 等. 基于人脸关键特征提取的表情识别[J]. 计算机工程, 2023, 49(2):254-262. RAN R S, WENG W W, WANG N, et al. Expression recognition based on the extraction of key facial features[J]. Computer Engineering, 2023, 49(2):254-262.(in Chinese)
[30] FARD A P, MAHOOR M H. Ad-Corre:adaptive correlation-based loss for facial expression recognition in the wild[J]. IEEE Access, 2022, 10:26756-26768.

选择文件类型/文献管理软件名称

选择包含的内容

基于关键区域遮挡与重建的人脸表情识别

Facial Expression Recognition Based on Key Region Masking and Reconstruction

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	李俊俊, 董建刚, 李坤. 基于Kubernetes的集群节能策略研究[J]. 计算机工程, 2024, 50(9): 82-91.
[2]	林畅, 郭伟, 任哲聪, 金海波. 基于Transformer的目标跟踪与分割统一算法[J]. 计算机工程, 2024, 50(9): 130-141.
[3]	李泽霖, 吕兆峰, 陈富强, 李克. 基于多跳信息融合的实体对齐模型[J]. 计算机工程, 2024, 50(9): 142-152.
[4]	王汝英, 马嘉骏, 董建强, 刘万龙, 张海涛, 尹凯, 赵博超. 基于MTS-BiGRU-DMHSA的工业负荷预测方法[J]. 计算机工程, 2024, 50(9): 169-178.
[5]	朱凯, 李理, 张彤, 江晟, 别一鸣. 基于Transformer的多阶段运动模糊图像修复网络[J]. 计算机工程, 2024, 50(9): 276-285.
[6]	张天鹏, 韩晶, 吕学强. 基于多任务学习的超分辨率辅助小目标检测[J]. 计算机工程, 2024, 50(9): 304-312.
[7]	郭敏, 张熙涵, 李阳. 融合注意力的教师互一致性半监督医学图像分割[J]. 计算机工程, 2024, 50(9): 313-323.
[8]	曾钰琦, 刘博, 钟柏昌, 钟瑾. 智慧教育下基于改进YOLOv8的学生课堂行为检测算法[J]. 计算机工程, 2024, 50(9): 344-355.
[9]	饶日昕, 王怡文, 曾砺志, 童心恬, 赵海涛. 面向废旧电缆检测的轻量化网络模型[J]. 计算机工程, 2024, 50(8): 22-30.
[10]	李华昱, 张智康, 闫阳, 岳阳. 基于知识图谱增强的领域多模态实体识别[J]. 计算机工程, 2024, 50(8): 31-39.
[11]	王蕾, 党时鹏, 潘丰. 基于卷积神经网络的隐匿性旁路预测模型[J]. 计算机工程, 2024, 50(8): 40-49.
[12]	陈瀚, 赵春蕾, 蒋昊达, 王春东. 基于融合模型与语义网络的App用户意图识别研究[J]. 计算机工程, 2024, 50(8): 50-63.
[13]	王夙喆, 张雪英, 陈晓玉, 李凤莲, 吴泽林. 基于有效注意力和GAN结合的脑卒中EEG增强算法[J]. 计算机工程, 2024, 50(8): 336-344.
[14]	王宇, 祁琦, 王纯, 许才. 储能变流器信号高精度故障诊断方法[J]. 计算机工程, 2024, 50(8): 389-396.
[15]	王炼红, 林飞鹏, 李潇瑶, 谌桂枝, 周莉. 融入课程知识图谱的KMAKT预测[J]. 计算机工程, 2024, 50(7): 23-31.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于关键区域遮挡与重建的人脸表情识别

Facial Expression Recognition Based on Key Region Masking and Reconstruction

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价