基于多任务学习的轻量级语音情感识别模型

doi:10.19678/j.issn.1000-3428.0064430

摘要/Abstract

摘要： 现有的语音情感识别（SER）模型存在训练参数量大、模型泛化性能差、情感识别准确率低等问题，利用有限的语音情感数据建立一个轻量级的模型以提高识别效率和准确率尤为重要。提出一种轻量级端到端多任务学习的P-CNN+Gender深度模型，该模型由语音特征组合网络、负责情感特征和性别特征提取的主体卷积网络以及情感和性别分类器组成。以语音的梅尔频率倒谱系数（MFCC）特征作为输入，特征组合网络使用多个大小不同的卷积核从MFCC特征中平行提取特征再进行组合，供后续的主体卷积网络进行情感特征和性别特征的提取。考虑到情感表达和性别的相关性，将性别分类作为辅助任务融合到情感分类中以提高模型的情感分类性能。实验结果表明，该模型在IEMOCAP、Emo-DB和CASIA语音情感数据集上的类别分类准确率分别达到73.3%、96.4%和93.9%，较P-CNN模型分别提高3.0、5.8和6.5个百分点，与3D-ACRNN、CNNBiRNN等模型相比，其训练参数量仅为其他模型的1/10~1/2,且处理速度更快、准确率更高。

关键词: 语音情感识别, MFCC特征, 特征提取, 卷积网络, 深度学习

Abstract: Current Speech Emotion Recognition（SER） models have shortcomings such as large numbers of training parameters，poor model generalization，and low emotion recognition accuracy. Therefore，under the condition of limited sample data，it is particularly important to build a lightweight model to improve model recognition efficiency and accuracy.To this end，this paper proposes a lightweight end-to-end multi-task deep learning model named P-CNN+Gender，which is composed of three parts：a speech feature combination network，body convolutional network responsible for emotion and gender feature extraction，and emotion and gender classifier.The model uses the Mel-Frequency Cepstral Coefficients（MFCC） features of speech as input，and the feature combination network uses convolutional kernels of different sizes to extract MFCC features in parallel and combine them for the subsequent body convolutional network to extract emotion and gender features.Finally，considering the correlation between emotional expression and gender，gender classification is integrated into emotion classification as an auxiliary task to improve the model's emotion classification performance.The model is tested on the IEMOCAP，Emo-DB，and CASIA speech emotion datasets and achieved Unweighted Accuracy（UA） results of 73.3%，96.4% and 93.9%，which are 3.0，5.8，and 6.5 percentage points higher than the P-CNN model，respectively.The training parameter quantity of this model is only 1/10-1/2 that of other models，such as 3D-ACRNN，CNNBiRNN，etc.，and the model achieves faster processing and higher accuracy.

Key words: Speech Emotion Recognition（SER）, Mel-Frequency Cepstral Coefficients（MFCC） feature, feature extraction, convolutional network, deep learning

中图分类号:

TP18

宋羽凯, 谢江. 基于多任务学习的轻量级语音情感识别模型[J]. 计算机工程, 2023, 49(5): 122-128.

SONG Yukai, XIE Jiang. Lightweight Speech Emotion Recognition Model Based on Multi-Task Learning[J]. Computer Engineering, 2023, 49(5): 122-128.

https://www.ecice06.com/CN/Y2023/V49/I5/122

图/表 20

20230515185459

20230515185502

20230515185506

20230515185509

20230515185515

20230515185518

20230515185521

20230515185525

20230515185529

20230515185536

20230515185540

20230515185544

20230515185547

20230515185551

20230515185555

20230515185559

20230515185603

20230515185607

20230515185612

20230515185616

参考文献

[1] SCHULLER B,STEIDL S,BATLINER A.The INTERSPEECH 2009 emotion challenge[C]//Proceedings of the 10th Annual Conference of the International Speech Communication Association.Brighton,UK:[s.n.],2009:312-315.
[2] SCHULLER B,STEIDL S,BATLINER A,et al.The INTERSPEECH 2013 computational paralinguistics challenge:social signals,conflict,emotion,autism[C]//Proceedings of the 14th Annual Conference of the International Speech Communication Association.Lyon,France:[s.n.],2013:148-512.
[3] RINGEVAL F,SCHULLER B,VALSTAR M,et al.AVEC 2015:the first affect recognition challenge bridging across audio,video,and physiological data[C]//Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge.New York,USA:ACM Press,2015:3-8.
[4] CHENCHAH F,LACHIRI Z.Acoustic emotion recognition using linear and nonlinear cepstral coefficients[J].International Journal of Advanced Computer Science and Applications,2015,6(11):1-10.
[5] SATT A,ROZENBERG S,HOORY R.Efficient emotion recognition from speech using deep learning on spectrograms[C]//Proceedings of the 18th Annual Conference of the International Speech Communication Association.Stockholm,Sweden:[s.n.],2017:1089-1093.
[6] ZHANG Y Y,DU J,WANG Z R,et al.Attention based fully convolutional network for speech emotion recognition[C]//Proceedings of Asia-Pacific Signal and Information Processing Association Annual Summit and Conference.Washington D.C.,USA:IEEE Press,2019:1771-1775.
[7] HUANG Z W,DONG M,MAO Q R,et al.Speech emotion recognition using CNN[C]//Proceedings of the 22nd ACM International Conference on Multimedia.New York,USA:ACM Press,2014:801-804.
[8] XU Y F,XU H,ZOU J Y.HGFM:a hierarchical grained and feature model for acoustic emotion recognition[C]//Proceedings of 2020 IEEE International Conference on Acoustics,Speech and Signal Processing.Washington D.C.,USA:IEEE Press,2020:6499-6503.
[9] DAI D Y,WU Z Y,LI R N,et al.Learning discriminative features from spectrograms using center loss for speech emotion recognition[C]//Proceedings of 2019 IEEE International Conference on Acoustics,Speech and Signal Processing.Washington D.C.,USA:IEEE Press,2019:7405-7409.
[10] SCHMIDT E M,KIM Y E.Learning emotion-based acoustic features with deep belief networks[C]//Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics.Washington D.C.,USA:IEEE Press,2011:65-68.
[11] HAN K,YU D,TASHEV I.Speech emotion recognition using deep neural network and extreme learning machine[C]//Proceedings of the 15th Annual Conference of the International Speech Communication Association.Singapore:[s.n.],2014:223-227.
[12] MIRSAMADI S,BARSOUM E,ZHANG C.Automatic speech emotion recognition using recurrent neural networks with local attention[C]//Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing.Washington D.C.,USA:IEEE Press,2017:2227-2231.
[13] CHEN M Y,HE X J,YANG J,et al.3-D convolutional recurrent neural networks with attention model for speech emotion recognition[J].IEEE Signal Processing Letters,2018,25(10):1440-1444.
[14] BUSSO C,BULUT M,LEE C C,et al.IEMOCAP:interactive emotional dyadic motion capture database[J].Language Resources and Evaluation,2008,42(4):335-359.
[15] BURKHARDT F,PAESCHKE A,ROLFES M,et al.A database of German emotional speech[C]//Proceedings of INTERSPEECH 2005.[S.l.]:ISCA,2005:1517-1520.
[16] LI P,SONG Y,LAN M,et al.An attention pooling based representation learning method for speech emotion recognition[C]//Proceedings of the 19th Annual Conference of the International Speech Communication Association.Hyderabad,India:[s.n.],2018:3087-3091.
[17] YENIGALLA P,KUMAR A,TRIPATHI S.Speech emotion recognition using spectrogram & phoneme embedding[C]//Proceedings of the 19th Annual Conference of the International Speech Communication Association.Hyderabad,India:[s.n.],2018:3688-3692.
[18] ZHAO J,MAO X,CHEN L.Speech emotion recognition using deep 1D & 2D CNN LSTM networks[J].Biomedical Signal Processing and Control,2019,47(4):312-323.
[19] McFEE B,RAFFEL C,LIANG D,et al.Librosa:audio and music signal analysis in Python[C]//Proceedings of the 14th Python in Science Conference.Austin,USA:SciPy,2015:18-25.
[20] LI R N,WU Z Y,JIA J,et al.Dilated residual network with multi-head self-attention for speech emotion recognition[C]//Proceedings of 2019 IEEE International Conference on Acoustics,Speech and Signal Processing.Washington D.C.,USA:IEEE Press,2019:6675-6679.
[21] ZHAO H,XIAO Y,HAN J,et al.Compact convolutional recurrent neural networks via binarization for speech emotion recognition[C]//Proceeding of 2019 IEEE International Conference on Acoustics,Speech and Signal Processing.Washington D.C.,USA:IEEE Press,2019:6690-6694.
[22] ZHONG Y,HU Y,HUANG H,et al.A lightweight model based on separable convolution for speech emotion recognition[C]//Proceedings of the 21st Annual Conference of the International Speech Communication Assoiation.Shanghai,China:[s.n.],2020:3331-3335.
[23] LATINUS M,TAYLOR M J.Discriminating male and female voices:differentiating pitch and gender[J].Brain Topography,2012,25(2):194-204.
[24] XU Z,MEYER P,FINGSCHEDT T.On the effects of speaker gender in emotion recognition training data[C]//Proceedings of ITG Conference 2018.Dresden,Germany:ITG-Fachbericht,2018:61-64.
[25] VOGT T,ANDRE E.Improving automatic emotion recognition from speech via gender differentiation[C]//Proceedings of Language Resources and Evaluation Conference.Genoa,Italy:LREC,2006:1123-1126.
[26] NEUMANN M,VU N T.Improving speech emotion recognition with unsupervised representation learning on unlabeled speech[C]//Proceedings of 2019 IEEE International Conference on Acoustics,Speech and Signal Processing.Washington D.C.,USA:IEEE Press,2019:7390-7394.

选择文件类型/文献管理软件名称

选择包含的内容