RRA-InceptionV3结合鲁棒稀疏表示的表情识别方法

doi:10.19678/j.issn.1000-3428.0064914

摘要/Abstract

摘要：

针对现实场景中人脸局部遮挡导致的表情识别准确度较低的问题，提出一种RRA-InceptionV3结合鲁棒稀疏表示的表情识别方法。将人脸图像通过多支路卷积运算和空洞卷积模块来获取丰富的表情特征，基于Asm-CBAM卷积注意力机制划分人脸表情特征的权重并进行多特征融合，随后堆叠密集残差模块，从多通道中自适应提取人脸特征信息，通过Asm-CBAM卷积注意力机制提高网络对人脸关键特征的注意力。在此基础上，利用鲁棒稀疏表示分类器方法对表情进行分类。在人脸数据集FER2013和CK+上的实验结果表明，该方法的人脸表情平均识别精度分别达到79.86%和98.74%，与OAD Net算法相比，分别高出7.50和3.14个百分点，能够高效提取人脸表情特征。此外，在人脸被遮挡的情况下具有较强的鲁棒性，有效提高了在人脸遮挡情况下表情识别的准确度。

关键词: 表情识别, 局部遮挡, 鲁棒稀疏表示分类器方法, 密集残差, Asm-CBAM模块, 空洞卷积

Abstract:

Facial expression recognition technology is an important subject in the field of computer vision.However, in real application, partial occlusion can lead to a sharp drop in the accuracy of facial expression recognition.In view of the low accuracy of facial expression recognition caused by partial occlusion of faces in real scenes, an RRA-InceptionV3 combined robust sparse representation method for expression recognition is proposed.First, the facial image is obtained through a multi-branch convolution operation and atrous convolution module to obtain rich expression features.the Asm-CBAM convolution attention mechanism then divides the weight of facial expression features.Multi-feature fusion and then stacking of dense residual blocks are used to adaptively extract face feature information from multiple channels.In addition, the Asm-CBAM convolutional attention mechanism is used again to improve the network's attention to key facial features.Finally, a Robust Sparse Representation Classifier(RSRC) method is used to classify expressions.The experimental results on face data sets FER2013 and CK+ show that the average recognition accuracy of the proposed method reaches 79.86% and 98.74%, respectively, which is 7.50 and 3.14 percentage points higher than that of the OAD Net algorithm, and can efficiently extract facial expression features.Moreover, it has strong robustness in the case of face occlusion, effectively improving the accuracy of expression recognition.

Key words: expression recognition, partial occlusion, Robust Sparse Representation Classifier(RSRC) method, dense residuals, Asm-CBAM module, atrous convolution

谢虹, 姜文刚. RRA-InceptionV3结合鲁棒稀疏表示的表情识别方法[J]. 计算机工程, 2023, 49(7): 196-203.

Hong XIE, Wengang JIANG. RRA-InceptionV3 Combined Robust Sparse Representation Method for Expression Recognition[J]. Computer Engineering, 2023, 49(7): 196-203.

https://www.ecice06.com/CN/Y2023/V49/I7/196

图/表 19

图1 基于RRA-InceptionV3结合鲁棒稀疏表示的表情识别方法框架

Fig.1 Framework of expression recognition method based on RRA-InceptionV3 combined with robust sparse representation

图2 InceptionV3网络结构

Fig.2 Structure of InceptionV3 network

图3 稀疏表示流程

Fig.3 Sparse representation procedure

图4 深度学习结合RSRC分类识别算法实现过程

Fig.4 Implementation process of deep learning combined with RSRC classification recognition algorithm

图5 RRDB结构

Fig.5 Structure of RRDB

图6 CBAM注意力机制示意图

Fig.6 Schematic diagram of CBAM attention mechanism

图7 Asm-CBAM注意力机制示意图

Fig.7 Schematic diagram of Asm-CBAM attention mechanism

图8 FER2013数据集示例

Fig.8 Example of FER2013 dataset

图9 CK+数据集示例

Fig.9 Example of CK+ dataset

表1 不同注意力模块的对比实验结果

Table 1 Comparative experimental results of different attention modules %

注意力模块	$ m_{\mathrm{mAP}} $
ScSE	90.22
CBAM	93.34
ECA	96.18
SE	91.53
Asm-CBAM	98.74

表2 残差模块的对比实验结果

Table 2 Comparative experimental results of residual modules %

残差模块类型	$ m_{\mathrm{mAP}} $
常规残差模块	98.60
倒置残差模块	97.40
瓶颈残差模块	98.54
密集残差模块	98.74

图10 RRDB和Ams-CBAM的融合结构

Fig.10 Fusion structure of RRDB and Ams-CBAM

表3 注意力模块对比实验结果

Table 3 Comparative experimental results of attention modules %

模块结构	$ m_{\mathrm{mAP}} $
结构A	98.32
结构B	98.74

表4 CK+数据集上不同遮挡部位表情识别结果

Table 4 Expression recognition results of different occluded parts on CK+ dataset %

方法	眼睛遮挡	嘴巴遮挡	鼻子遮挡
NN	95.20	84.09	97.31
Baysian	94.85	87.20	97.04
SRC	95.74	92.63	97.63
SVM	93.11	89.57	96.85
RSRC	96.51	94.22	98.49

图11 CK+数据集上不同遮挡程度表情识别结果

Fig.11 Expression recognition results of different occlusion degree on CK+ dataset

表5 消融实验结果

Table 5 Ablation experiment results

编号	RSRC	RRDB	Asm-CBAM	m_mAP/%
编号	RSRC	RRDB	Asm-CBAM	FER2013	CK+
1	▌	▌	×	75.88	98.10
2	▌	▌	▌	76.32	98.74
3	×	▌	▌	75.48	97.92
4	▌	×	×	74.53	96.22
5	×	×	▌	73.77	95.98
6	▌	×	▌	74.66	96.98
7	×	▌	×	72.10	95.64
8	×	×	×	71.11	94.32

表6 FER2013数据集上的实验结果

Table 6 Experimental results on the FER2013 dataset %

方法	愤怒	厌恶	恐惧	快乐	悲伤	惊喜	中立	藐视	总体
文献[21]方法	66.32	78.54	58.22	92.91	62.43	90.59	75.81	66.52	75.86
文献[22]方法	64.89	75.68	56.70	92.10	58.31	98.99	75.48	67.35	72.36
文献[23]方法	65.19	76.22	55.45	90.10	63.88	90.69	75.33	66.62	72.11
文献[24]方法	64.34	74.65	52.77	87.07	59.52	87.41	74.25	65.59	71.93
文献[25]方法	65.98	75.75	55.30	92.81	62.90	89.66	76.99	67.20	77.18
本文方法	68.37	80.16	60.10	94.17	65.23	90.44	78.50	70.65	79.86

表7 CK+数据集上实验结果

Table 7 Experimental results on the CK+ dataset %

方法	愤怒	厌恶	恐惧	快乐	悲伤	惊喜	中立	藐视	总体
文献[21]方法	96.95	95.27	96.46	100.00	97.20	99.00	97.14	94.27	97.68
文献[22]方法	93.45	93.67	93.92	95.34	94.88	97.19	94.45	94.23	95.60
文献[23]方法	96.10	94.27	94.64	98.99	96.05	99.03	96.89	94.11	97.10
文献[24]方法	96.88	94.82	95.88	99.67	96.54	99.07	97.03	93.64	96.19
文献[25]方法	95.30	93.77	92.98	95.50	94.90	97.22	94.50	94.01	96.76
本文方法	97.34	95.33	96.70	100.00	97.55	100.00	97.54	95.20	98.74

图12 本文方法的可视化结果

Fig.12 Visualization results of method in this paper

参考文献 25

1	XU Q T, ZHAO N J. A facial expression recognition algorithm based on CNN and LBP feature[C]//Proceedings of the 4th IEEE Information Technology, Networking, Electronic and Automation Control Conference. Washington D. C., USA: IEEE Press, 2020: 2304-2308.
2	WANG Y, WU Q L. Research on face recognition technology based on PCA and SVM[C]//Proceedings of the 7th International Conference on Big Data Analytics. Washington D. C., USA: IEEE Press, 2022: 248-252.
3	LI Y, MA H, FANG C X, et al. Fusion of PHOG and Gabor for facial expression recognition[C]//Proceedings of the 7th International Conference on Intelligent Computing and Signal Processing. Washington D. C., USA: IEEE Press, 2022: 968-971.
4	ZHOU Y, FENG Y Y, ZENG S Y, et al. Facial expression recognition based on convolutional neural network[C]//Proceedings of the 10th IEEE International Conference on Software Engineering and Service Science. Washington D. C., USA: IEEE Press, 2020: 410-413.
5	LESIANGI F S, MAUKO A Y, DJAHI B S. Feature extraction Hue, Saturation, Value(HSV) and Gray Level Cooccurrence Matrix(GLCM) for identification of woven fabric motifs in south central Timor regency. Journal of Physics Series, 2021, (1): 012010.
6	THAMBA M W, HEMAJOTHI S, MARY A E A. Real-time facial expression recognition for affect identification using multi-dimensional SVM. Journal of Ambient Intelligence and Humanized Computing, 2021, 12 (6): 6355- 6365. doi: 10.1007/s12652-020-02221-6
7	FAN Z, XIE J K, WANG Z Y, et al. Image classification method based on improved KNN algorithm. Journal of Physics Series, 2021, 30 (3): 012009.
8	WANG S Q, GUAN S E, LIN H, et al. Micro-expression recognition based on optical flow and PCANet. Sensors, 2022, 22 (11): 4296. doi: 10.3390/s22114296
9	ZHANG L, XU C, LI S. Facial expression recognition of infants based on multi-stream CNN fusion network[C]// Proceedings of the 5th IEEE International Conference on Signal and Image Processing. Washington D. C., USA: IEEE Press, 2020: 37-41.
10	ZHANG Q. Facial expression recognition in VGG network based on LBP feature extraction[C]//Proceedings of the 5th International Conference on Mechanical, Control and Computer Engineering. Washington D. C., USA: IEEE Press, 2021: 2089-2092.
11	BOUALLEGUE G, DJEMAL R. EEG person identification using facenet, LSTM-RNN and SVM[C]//Proceedings of the 17th International Multi-Conference on Systems, Signals & Devices. Washington D. C., USA: IEEE Press, 2021: 22-28.
12	WANG X P, LI W X, HUANG D. Expression-latent-space-guided GAN for facial expression animation based on discrete labels[C]//Proceedings of the 16th IEEE International Conference on Automatic Face and Gesture Recognition. Washington D. C., USA: IEEE Press, 2022: 1-8.
13	CHEN Y, LIU S. Deep partial occlusion facial expression recognition via improved CNN[C]//Proceedings of International Conference on Visual Computing. Berlin, Germany: Springer, 2020: 451-462.
14	XU X K, TAO R, FENG X Y, et al. A lightweight facial expression recognition network based on dense connections[C]//Proceedings of International Conference on Knowledge Management in Organizations. Berlin, Germany: Springer, 2022: 347-359.
15	XU X F, ZHANG L, DUAN C D, et al. Research on inception module incorporated Siamese convolutional neural networks to realize face recognition. IEEE Access, 2019, 8, 12168- 12178.
16	NAINAN N A, JALAN R. Real time face mask detection using MobileNetV2 and InceptionV3 models[C]//Proceedings of IEEE Mysore Sub Section International Conference. Washington D. C., USA: IEEE Press, 2021: 341-345.
17	ALI W, TIAN W H, DIN S U, et al. Classical and modern face recognition approaches: a complete review. Multimedia Tools and Applications, 2021, 80 (3): 4825- 4880. doi: 10.1007/s11042-020-09850-1
18	WRIGHT J, YANG A Y, GANESH A, et al. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31 (2): 210- 227. doi: 10.1109/TPAMI.2008.79
19	WANG W, JIN J X. Super-resolution reconstruction of human face image based on network of residual dense connection channel attentional[C]//Proceedings of the 5th International Conference on Mechanical, Control and Computer Engineering. Washington D. C., USA: IEEE Press, 2021: 1972-1977.
20	ZHANG X, CHEN Z, WEI Q J. Research and application of facial expression recognition based on attention mechanism[C]//Proceedings of IEEE Asia-Pacific Conference on Image Processing, Electronics and Computers. [S. 1. ]: IEEE Press, 2021: 282-285.
21	LI J, JIN K, ZHOU D L, et al. Attention mechanism-based CNN for facial expression recognition. Neurocomputing, 2020, 411, 340- 350. doi: 10.1016/j.neucom.2020.06.014
22	GERA D, BALASUBRAMANIAN S, JAMI A. CERN: compact facial expression recognition net. Pattern Recognition Letters, 2022, 155, 9- 18. doi: 10.1016/j.patrec.2022.01.013
23	DING H, ZHOU P, CHELLAPPA R. Occlusion-adaptive deep network for robust facial expression recognition[C]//Proceedings of IEEE International Joint Conference on Biometrics. Washington D. C., USA: IEEE Press, 2021: 1-9.
24	XIE X Y, MA Y A, LIU B, et al. A deep-learning-based real-time detector for grape leaf diseases using improved convolutional neural networks. Frontiers in Plant Science, 2020, 11, 751. doi: 10.3389/fpls.2020.00751
25	ZHANG J W, YAN X D, CHENG Z L, et al. A face recognition algorithm based on feature fusion. Concurrency and Computation, 2022, 34 (14): e5748.

[1]	陈晓玉, 沈晨, 沈阅, 孔德明. 基于改进SwiftNet的堆场图像实时分割网络[J]. 计算机工程, 2024, 50(6): 296-303.
[2]	李晶, 李健, 陈海丰, 张倩, 王丽燕, 裴二成. 基于关键区域遮挡与重建的人脸表情识别[J]. 计算机工程, 2024, 50(5): 241-249.
[3]	杨瑞君, 秦晋京, 程燕. 基于生成对抗网络的自然场景低照度增强模型[J]. 计算机工程, 2024, 50(1): 279-288.
[4]	孙龙, 张荣芬, 刘宇红, 饶庭漓. 监控视角下密集人群口罩佩戴检测算法[J]. 计算机工程, 2023, 49(9): 313-320.
[5]	刘志浩, 孟凡云, 王金鹤, 张楠. 基于空洞卷积与注意力模块的立体匹配算法[J]. 计算机工程, 2023, 49(8): 223-231.
[6]	丰芳宇, 罗晓曙, 蒙志明, 王广宇. 基于抗混叠残差注意力网络的人脸表情识别[J]. 计算机工程, 2023, 49(8): 190-198.
[7]	杨长沛, 廖列法. 基于门控空洞卷积特征融合的中文命名实体识别[J]. 计算机工程, 2023, 49(8): 85-95.
[8]	谌雨章, 黄逸姿, 张钧涵. 基于多速率空洞卷积的多尺度水下小目标检测[J]. 计算机工程, 2023, 49(6): 257-264.
[9]	冉瑞生, 翁稳稳, 王宁, 彭顺顺. 基于人脸关键特征提取的表情识别[J]. 计算机工程, 2023, 49(2): 254-262.
[10]	于明, 钟元想, 王岩. 人脸微表情分析方法综述[J]. 计算机工程, 2023, 49(2): 1-14.
[11]	张子珺, 陈劲松, 钱夕元. 基于改进条件生成对抗网络的书法字骨架提取[J]. 计算机工程, 2023, 49(10): 272-279.
[12]	蓝峥杰, 王烈, 聂雄. 一种基于词频‐逆文档频率和混合损失的表情识别算法[J]. 计算机工程, 2023, 49(1): 295-302,310.
[13]	窦允冲, 侯进, 曾雷鸣, 陈子锐. 基于反馈机制与空洞卷积的道路小目标检测网络[J]. 计算机工程, 2023, 49(1): 287-294.
[14]	刘腾, 刘宏哲, 李学伟, 徐成. 基于无锚框分割网络改进的实例分割方法[J]. 计算机工程, 2022, 48(9): 239-247,253.
[15]	郑伟鹏, 罗晓曙, 蒙志明. 基于改进轻量级秩扩展网络的人脸表情识别方法[J]. 计算机工程, 2022, 48(9): 189-196.

选择文件类型/文献管理软件名称

选择包含的内容