Facial Expression Recognition Model Based on Convolutional Neural Network with Occlusion Perception

doi:10.19678/j.issn.1000-3428.0059166

Abstract

Abstract: To reduce the difficulty in extracting features of an occluded face, a dual-channel Convolutional Neural Network (CNN) model with occlusion perception is proposed.The model is constructed by integrating newly designed occlusiondecision units into VGG16 network, which aims at extractingexpression-related features of the areas that are less occluded.The model employs the transfer learning algorithm to pre-train the parameters of the convolutional layer, which means to alleviate the over-fittingproblem.At the meantime, the expression-related features of the whole facial image are extracted by the modified residual network.Finally, the outputs of theperceptive neural network and residual network arefused in a weighted manner.The experimental results show that the proposed model achieves an accuracy of 97.33% on CK+, 86% on RAF-DB, and 61.06%on SFEW.Compared with traditional OPCNN, ResNet, and VGG16 models, the proposed model exhibits a significant improvement in the accuracy of recognizing the expression of an occluded face.

Key words: Convolutional Neural Network(CNN), facial expression recognition, transfer learning, feature fusion, residual network

摘要： 针对面部遮挡情况下表情特征难以提取的问题，提出一种双通道遮挡感知神经网络模型。设计区域遮挡判定单元并集成到VGG16网络中形成遮挡感知神经网络，提取面部图像中未遮挡区域及遮挡较少区域的表情特征。运用迁移学习算法对卷积层参数进行预训练，减轻训练数据样本不足带来的过拟合问题。通过优化残差网络提取全脸表情相关特征，在此基础上加权融合遮挡感知神经网络和残差网络的输出以识别表情。在CK+、RAF-DB、SFEW这3个公开数据库上进行对比实验，结果表明，该模型平均准确率分别达到97.33%、86%、61.06%，与OPCNN、ResNet、VGG16等传统卷积神经网络模型相比，有效提高了面部遮挡情况下的表情识别精度。

关键词: 卷积神经网络, 面部表情识别, 迁移学习, 特征融合, 残差网络

CLC Number:

TP311.1

WANG Jun, ZHAO Kai, CHENG Yong. Facial Expression Recognition Model Based on Convolutional Neural Network with Occlusion Perception[J]. Computer Engineering, 2021, 47(10): 242-251.

王军, 赵凯, 程勇. 基于遮挡感知卷积神经网络的面部表情识别模型[J]. 计算机工程, 2021, 47(10): 242-251.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0059166

http://www.ecice06.com/EN/Y2021/V47/I10/242

Figures/Tables 14

References

[1] FARFADE S S, SABERIAN M, LI L J.Multi-view face detection using deep convolutional neural networks[C]//Proceedings of the 5th International Conference on Multimedia Retrieval.New York, USA:ACM Press, 2005:643-650.
[2] ZHANG K P, ZHANG Z P, LI Z F, et al.Joint face detection and alignment using multitask cascaded convolutional networks[J].IEEE Signal Processing Letters, 2016, 23(10):1499-503.
[3] WANG P R, CHE W J, BO X.A cascaded framework for model-based 3D face reconstruction[C]//Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing.Washington D.C., USA:IEEE Press, 2018:3151-3155.
[4] BURGOS-ARTIZZU X P, FLEUREAU J, DUMAS O, et al.Real-time expression-sensitive HMD face reconstruction[M].New York, USA:ACM Press, 2015:4-13.
[5] DOU P F, SHAH S K, KAKADIARIS I A.End-to-end 3D face reconstruction with deep neural networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:1503-1512.
[6] LUCEY P, COHN J F, T.KANADE T, et al.The extended cohn-kanade dataset:a complete dataset for action unit and emotion-specified expression[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2010:94-101.
[7] LYONS M, AKAMATSU S, KAMACHI M, et al.Coding facial expressions with gabor wavelets[M].Washington D.C., USA:IEEE Press, 1998.
[8] PANTIC M, VALSTAR M, RADEMAKER R, et al.Web-based database for facial expression analysis[C]//Proceedings of IEEE International Conference on Multimedia and Expo.Washington D.C., USA:IEEE Press, 2005:5-12.
[9] DHALL A, RAMANA MURTHY O V, GOECKE R, et al.Video and image based emotion recognition challenges in the wild:emotiw 2015[C]//Proceedings of International Conference on Multimodal Interaction.New York, USA:ACM Press, 2015:423-426.
[10] GOODFELLOW I J, ERHAN D, CARRIER P, et al.challenges in representation learning:a report on three machine learning contests[J].Neural Networks, 2015, 64(1):59-63.
[11] DING H, ZHOU S H K, CHELLAPPA R.Facenet2expnet:regularizing a deep face recognition net for expression recognition[C]//Proceedings of the 12th IEEE International Conference on Automatic Face & Gesture Recognition.Washington D.C., USA:IEEE Press, 2017:118-126.
[12] JUNG H, LEE S, YIM J, et al.Joint fine-tuning in deep neural networks for facial expression recognition[C]//Proceedings of IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2015:2983-2991.
[13] LIU P, HAN S Z, MENG Z B, et al.Facial expression recognition via a boosted deep belief network[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2014:1805-1812.
[14] KRIZHEVSKY A, SUTSKEVER I, HINTON G E, et al.ImageNet classification with deep convolutional neural networks[EB/OL].[2020-07-01].https://users.ics.aalto.fi/perellm1/thesis/summaries_html/node64.html.
[15] SIMONYAN K, ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].[2020-07-03], http://arXiv:1409.1556.
[16] SZEGEDY C, LIU W, JIA Y Q, et al.Going deeper with convolutions[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:1-9.
[17] MAGGIORI E, TARABAIKA Y, CHARPIAT G, et al.Convolutional neural networks for large-scale remote-sensing image classification[J].IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(2):645-657.
[18] SLAVKOVIKJ V, VERSTOCKT S, NEVE W D, et al.Hyperspectral image classification with convolutional neural networks[C]//Proceedings of the 23rd ACM International Conference.New York, USA:ACM press, 2015:1159-1162.
[19] REDMON J, DIVVALA S, GIRSHICK R, et al.You only look once:unified, real-time object detection[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:779-788.
[20] REN S P, HE K M, GIRSHICK R, et al.Faster R-CNN:towards real-time object detection with region proposal networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(5):1137-1149.
[21] ZEILER M D, FERGUS R.Visualizing and understanding convolutional networks[C]//Proceedings of Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2014:818-833.
[22] HUANG G, LIU Z, MAATEN L V, et al.Densely connected convolutional networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:2261-2269.
[23] YANG B, CAO J M, NI R R, et al.Facial expression recognition using weighted mixture deep neural network based on double-channel facial images[J].IEEE Access, 2017, 6:4630-4640.
[24] LI Y, ZENG J B, SHAN S G, et al.Occlusion aware facial expression recognition using CNN with attention mechanism[J].IEEE Transactions on Image Processing, 2018, 28(5):2439-2450.
[25] SHAN L, DENG W H, DU J P.Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:2584-2593.
[26] LOPES A T, AGUIAR E, SOUZA A F, et al.Facial expression recognition with convolutional neural networks:coping with few data and the training sample order[J].Pattern Recognition, 2017, 61(1):610-628.
[27] YU Z D, ZHANG C.Image based static facial expression recognition with multiple deep network learning[C]//Proceedings of 2015 ACM International Conference on Multimodal Interaction.New York, NY:ACM Press, 2015:435-442.
[28] JIE S, QIAN Y S.Three convolutional neural network models for facial expression recognition in the wild[J].Neurocomputing, 2019, 355(1):82-92.
[29] MOLLAHOSSEINI A, CHAN D, MAHOOR M H.Going deeper in facial expression recognition using deep neural networks[C]//Proceedings of 2016 IEEE Winter Conference on Applications of Computer Vision.Washington D.C., USA:IEEE Press, 2016:1-10.
[30] ISOLA P, ZHU J Y, ZHOU T H, et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:5967-5976.
[31] MENG Z B, LIU P, CAI J, et al.Identity-aware convolutional network for facial expression recognition[C]//Proceedings of the 12th IEEE International Conference on Automatic Face and Gesture Recognition.Washington D.C., USA:IEEE Press, 2017:558-565.
[32] NG H W, NGUYEN D V, VONIKAKIS V, et al.Deep Learning for Emotion Recognition on Small Datasets Using Transfer Learning[C]//Proceedings of the 2015 ACM International Conference on Multimodal Interaction.New York, USA:ACM Press, 2015:443-449.

Please choose a citation manager

Content to export