基于异构并行神经网络的语音情感识别

doi:10.19678/j.issn.1000-3428.0061076

摘要/Abstract

摘要： 提取能表征语音情感的特征并构建具有较强鲁棒性和泛化性的声学模型是语音情感识别系统的核心。面向语音情感识别构建基于注意力机制的异构并行卷积神经网络模型AHPCL，采用长短时记忆网络提取语音情感的时间序列特征，使用卷积操作提取语音空间谱特征，通过将时间信息和空间信息相结合共同表征语音情感，提高预测结果的准确率。利用注意力机制，根据不同时间序列特征对语音情感的贡献程度分配权重，实现从大量特征信息中选择出更能表征语音情感的时间序列。在CASIA、EMODB、SAVEE等3个语音情感数据库上提取音高、过零率、梅尔频率倒谱系数等低级描述符特征，并计算这些低级描述符特征的高级统计函数共得到219维的特征作为输入进行实验验证。结果表明，AHPCL模型在3个语音情感数据库上分别取得了86.02%、84.03%、64.06%的未加权平均召回率，相比LeNet、DNN-ELM和TSFFCNN基线模型具有更强的鲁棒性和泛化性。

关键词: 语音情感识别, 谱特征, 韵律特征, 注意力机制, 异构并行分支, 循环神经网络

Abstract: The core of a Speech Emotion Recognition(SER) system is to extract features that can best represent speech emotion and construct an acoustic model with strong robustness and generalization.In this study, a heterogeneous parallel Recurrent Neural Network(RNN) model based on the attention mechanism AHPCL is constructed for SER.The Long Short-Term Memory(LSTM) network is used to extract the time-series features of speech emotion, and the convolution operation is used to extract the speech spatial spectral features.By combining temporal and spatial information to jointly represent speech emotion, the accuracy of the prediction results is improved.The attention mechanism is used to assign weights according to the contribution of different time-series features to speech emotion to select a time sequence that better represents speech emotion from a large amount of feature information.Low-level descriptor features such as pitch, Zero Crossing Rate(ZCR), and Mel-Frequency Cepstrum Coefficient(MFCC) are extracted from three speech emotion databases, namely CASIA, EMODB, and SAVEE, and the high-level statistical functions of these low-level descriptor features are calculated to obtain 219 dimensional features.The experimental results show that the proposed model achieves 86.02%, 84.03%, and 64.06% Unweighted Average Recall(UAR) on the CASIA, EMODB, and SAVEE databases, respectively.Compared with the LeNet, DNN-ELM, and TSFFCNN baseline models, the AHPCL model exhibits greater robustness and generalization.

Key words: Speech Emotion Recognition(SER), spectral feature, prosodic feature, attention mechanism, heterogeneous parallel branch, Recurrent Neural Network(RNN)

中图分类号:

TP183

张会云, 黄鹤鸣. 基于异构并行神经网络的语音情感识别[J]. 计算机工程, 2022, 48(4): 113-118.

ZHANG Huiyun, HUANG Heming. Speech Emotion Recognition Based on Heterogeneous Parallel Neural Network[J]. Computer Engineering, 2022, 48(4): 113-118.

https://www.ecice06.com/CN/Y2022/V48/I4/113

图/表 7

20230131200933

20230131200936

20230131200940

20230131200943

20230131200947

20230131200950

20230131200953

参考文献

[1] TRIANTO R, TAI T C, WANG J C.Fast-LSTM acoustic model for distant speech recognition[C]//Proceedings of 2018 IEEE International Conference on Consumer Electronics.Washington D.C., USA:IEEE Press, 2018:1-4.
[2] OCQUAYE E N N.Speech emotion recognition via domain adaptation[D].Zhenjiang:Jiangsu University, 2020.
[3] 张鹤鹏.基于表情和语音信号的情感识别研究[D].济南:山东大学, 2020. ZHANG H P.Research on emotion recognition based on expression and speech signal[D].Jinan:Shandong University, 2020.(in Chinese)
[4] 张昕然.跨库语音情感识别若干关键技术研究[D].南京:东南大学, 2016. ZHANG X R.Research on several key technologies in cross-corpus speech emotion recognition[D].Nanjing:Southeast University, 2016.(in Chinese)
[5] TZIRAKIS P, ZHANG J H, SCHULLER B W.End-to-end speech emotion recognition using deep neural networks[C]//Proceedings of 2018 IEEE International Conference on Acoustics, Speech and Signal Processing.Washington D.C., USA:IEEE Press, 2018:5089-5093.
[6] LIAN Z, LIU B, TAO J H.CTNet:conversational transformer network for emotion recognition[J].IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29(1):985-1000.
[7] DENG J, XU X Z, ZHANG Z X, et al.Semisupervised autoencoders for speech emotion recognition[J].IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2018, 26(1):31-43.
[8] 高帆, 张雪英, 黄丽霞, 等.基于DBM-LSTM的多特征语音情感识别[J].计算机工程与设计, 2020, 41(2):465-470. GAO F, ZHANG X Y, HUANG L X, et al.Multi-feature speech emotion recognition based on DBM-LSTM[J].Computer Engineering and Design, 2020, 41(2):465-470.(in Chinese)
[9] 宋春晓.情感语音的非线性特征提取及特征优化的研究[D].太原:太原理工大学, 2018. SONG C X.Research on nonlinear feature extraction and feature optimization of emotional speech[D].Taiyuan:Taiyuan University of Technology, 2018.(in Chinese)
[10] GUO L L, WANG L B, DANG J W, et al.A feature fusion method based on extreme learning machine for speech emotion recognition[C]//Proceedings of 2018 IEEE International Conference on Acoustics, Speech and Signal Processing.Washington D.C., USA:IEEE Press, 2018:2666-2670.
[11] MIRSAMADI S, BARSOUM E, ZHANG C.Automatic speech emotion recognition using recurrent neural networks with local attention[C]//Proceedings of 2017 IEEE International Conference on Acoustics, Speech and Signal Processing.Washington D.C., USA:IEEE Press, 2017:2227-2231.
[12] ZHANG T, ZHENG W M, CUI Z, et al.Spatial-temporal recurrent neural network for emotion recognition[J].IEEE Transactions on Cybernetics, 2019, 49(3):839-847.
[13] TAO F, LIU G.Advanced LSTM:a study about better time dependency modeling in emotion recognition[C]//Proceedings of 2018 IEEE International Conference on Acoustics, Speech and Signal Processing.Washington D.C., USA:IEEE Press, 2018:2906-2910.
[14] PENG Z C, ZHU Z, UNOKI M, et al.Auditory-inspired end-to-end speech emotion recognition using 3D convolutional recurrent neural networks based on spectral-temporal representation[C]//Proceedings of 2018 IEEE International Conference on Multimedia and Expo.Washington D.C., USA:IEEE Press, 2018:1-6.
[15] SHU X B, ZHANG L Y, SUN Y L, et al.Host-parasite:graph LSTM-in-LSTM for group activity recognition[J].IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(2):663-674.
[16] MAO Q R, DONG M, HUANG Z W, et al.Learning salient features for speech emotion recognition using convolutional neural networks[J].IEEE Transactions on Multimedia, 2014, 16(8):2203-2213.
[17] XIE Y, LIANG R Y, LIANG Z L, et al.Speech emotion classification using attention-based LSTM[J].IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019, 27(11):1675-1685.
[18] 姜芃旭, 傅洪亮, 陶华伟, 等.一种基于卷积神经网络特征表征的语音情感识别方法[J].电子器件, 2019, 42(4):998-1001. JIANG P X, FU H L, TAO H W, et al.Feature characterization based on convolution neural networks for speech emotion recognition[J].Chinese Journal of Electron Devices, 2019, 42(4):998-1001.(in Chinese)
[19] YI L, MAK M W.Improving speech emotion recognition with adversarial data augmentation network[EB/OL].[2021-02-14].https://www.researchgate.net/publication/346055097_Improving_Speech_Emotion_Recognition_With_Adversarial_Data_Augmentation_Network.
[20] WU M, SU W J, CHEN L, et al.Two-stage fuzzy fusion based-convolution neural network for dynamic emotion recognition[EB/OL].[2021-02-14].https://www.researchgate.net/publication/338564019_Two-stage_Fuzzy_Fusion_based-Convolution_Neural_Network_for_Dynamic_Emotion_Recognition.
[21] LIU Z T, XIE Q, WU M, et al.Speech emotion recognition based on an improved brain emotion learning model[J].Neurocomputing, 2018, 309:145-156.
[22] SUN Y X, WEN G H, WANG J B.Weighted spectral features based on local Hu moments for speech emotion recognition[J].Biomedical Signal Processing and Control, 2015, 18(1):80-90.
[23] TAO H W, LIANG R Y, ZHA C, et al.Spectral features based on local Hu moments of Gabor spectrograms for speech emotion recognition[J].IEICE Transactions on Information and Systems, 2016, E99-D(8):2186-2189.
[24] 缪裕青, 邹巍, 刘同来, 等.基于参数迁移和卷积循环神经网络的语音情感识别[J].计算机工程与应用, 2019, 55(10):135-140, 198. MIAO Y Q, ZOU W, LIU T L, et al.Speech emotion recognition model based on parameter transfer and convolutional recurrent neural network[J].Computer Engineering and Applications, 2019, 55(10):135-140, 198.(in Chinese)
[25] CHEN L, SU W J, WU M, et al.A fuzzy deep neural network with sparse autoencoder for emotional intention understanding in human-robot interaction[J].IEEE Transactions on Fuzzy Systems, 2020, 28(7):1252-1264.
[26] WEN G, LI H, HUANG J, et al.Random deep belief networks for recognizing emotions from speech signals[EB/OL].[2021-02-14].http://europepmc.org/article/PMC/5357547.
[27] CHEN M Y, HE X J, YANG J, et al.3-D convolutional recurrent neural networks with attention model for speech emotion recognition[J].IEEE Signal Processing Letters, 2018, 25(10):1440-1444.
[28] 乔栋, 陈章进, 邓良, 等.改进语音处理的卷积神经网络中文语音情感识别[J].计算机工程, 2022, 48(2):281-290. QIAO D, CHEN Z J, DENG L, et al.Speech emotion recognition based on improved speech processing and convolution neural network[J].Computer Engineering, 2022, 48(2):281-290.(in Chinese)
[29] NOROOZI F, MARJANOVIC M, NJEGUS A, et al.Audio-visual emotion recognition in video clips[J].IEEE Transactions on Affective Computing, 2019, 10(1):60-75.

选择文件类型/文献管理软件名称

选择包含的内容