Method for Conversational Emotion Recognition Using Hierarchical R-GCN

doi:10.19678/j.issn.1000-3428.0060346

Computer Engineering ›› 2022, Vol. 48 ›› Issue (1): 85-92. doi: 10.19678/j.issn.1000-3428.0060346

• Artificial Intelligence and Pattern Recognition • Previous Articles Next Articles

Method for Conversational Emotion Recognition Using Hierarchical R-GCN

LAI Helang^1,2, LI Lingli², HU Wanling¹, YAN Xueming³

1. School of Computer Science, South China Normal University, Guangzhou 510631, China;
2. Department of Information Management, Guangdong Justice Police Vocational College, Guangzhou 510520, China;
3. School of Information Science and Technology, Guangdong University of Foreign Studies, Guangzhou 510006, China

Received:2020-12-21 Revised:2021-01-21 Published:2021-02-03

一种基于层次化R-GCN的会话情绪识别方法

赖河蒗^1,2, 李玲俐², 胡婉玲¹, 颜学明³

1. 华南师范大学计算机学院, 广州 510631;
2. 广东司法警官职业学院信息管理系, 广州 510520;
3. 广东外语外贸大学信息科学与技术学院, 广州 510006

作者简介:赖河蒗(1985-),男,博士研究生,主研方向为情感分析、服务计算理论与技术;李玲俐(通信作者),教授;胡婉玲,硕士研究生;颜学明,博士。
基金资助:
国家自然科学基金青年基金项目（62006053）；广东省教育厅特色创新类项目（2018KQNCX072）；2020年度广东省普通高校青年创新人才项目（2020KQNCX186）；2020年度广东省普通高校特色创新项目（2020KTSCX273）；广东省高等教育学会“十三五”规划2019年度高校青年教师高等教育学研究课题（19GGZ070）；广东司法警官职业学院第四届院级课题（2020YB16）。

Abstract

Abstract: Conversational emotion recognition should consider not only the emotions of the speakers themselves, but also the emotions passing between speakers.This paper proposes an emotion recognition method based on Hierarchical Relational Graph Convolutional Network(HRGCN), which considers both two types of emotions to improve the recognition performance.The method employs a Basic Neural Network(BNN) to optimize the feature data of the conversational sequence, and divides the sequence into two different conversational subsequences according to the speaker.Two local Relational Graph Convolutional Networks(R-GCN) are used for local modelling of these two subsequences respectively, and the two locally modeled subsequences are reconcatenated in chronological order of the conversation.Furthermore, the global R-GCN is used to model the reconcatenated sequence globally.through hierarchical modeling of the input multimodal feature data, HRGCN can capture more contextual information.The experimental results on the IEMOCAP dataset show that HRGCN displays an accuracy of 84.48% and a F1 score of 84.40%, higher than LSTM, GRU and other mainstream recurrent neural networks.

Key words: Basic Neural Network(BNN), Relational Graph Convolutional Network(R-GCN), conversation, emotion recognition, artificial intelligence

摘要： 会话情绪识别包括说话者自身情绪影响以及说话者之间情绪影响这2个重要因素，为了充分考虑上述影响因素以提高会话情绪识别效果，提出一种基于层次化关系图卷积神经网络（HRGCN）的识别方法。使用一个基础神经网络对会话序列的特征数据进行优化，按照不同的说话者划分出2个不同的会话子序列，采用2个局部关系图卷积神经网络（R-GCN）分别对2个子序列进行局部建模，按照会话发生的时间顺序重新整合局部建模后的2个子序列，并利用全局R-GCN对其进行全局建模。通过对输入的多模态特征数据的分层次建模，使得会话序列捕获到更多的上下文信息。在IEMOCAP数据集上的实验结果表明，与当前流行的循环神经网络LSTM、GRU等相比，HRGCN方法的会话情绪识别性能较高，准确率与F1值分别达到84.48%与84.40%。

关键词: 基础神经网络, 关系图卷积神经网络, 会话, 情绪识别, 人工智能

CLC Number:

TP183

LAI Helang, LI Lingli, HU Wanling, YAN Xueming. Method for Conversational Emotion Recognition Using Hierarchical R-GCN[J]. Computer Engineering, 2022, 48(1): 85-92.

赖河蒗, 李玲俐, 胡婉玲, 颜学明. 一种基于层次化R-GCN的会话情绪识别方法[J]. 计算机工程, 2022, 48(1): 85-92.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0060346

http://www.ecice06.com/EN/Y2022/V48/I1/85

Figures/Tables 9

References

[1] 牛耘, 潘明慧, 魏欧, 等.基于词典的中文微博情绪识别[J].计算机科学, 2014, 41(9):253-258, 289. NIU Y, PAN M H, WEI O, et al.Emotion analysis of Chinese microblogs using lexicon-based approach[J].Computer Science, 2014, 41(9):253-258, 289.(in Chinese)
[2] 周锦峰, 叶施仁, 王晖.基于深度卷积神经网络模型的文本情感分类[J].计算机工程, 2019, 45(3):300-308. ZHOU J F, YE S R, WANG H.Text sentiment classification based on deep convolutional neural network model[J].Computer Engineering, 2019, 45(3):300-308.(in Chinese)
[3] 蔡林森, 彭超, 陈思远, 等.基于多样化特征卷积神经网络的情感分析[J].计算机工程, 2019, 45(4):169-174, 180. CAI L S, PENG C, CHEN S Y, et al.Sentiment analysis based on multiple features convolutional neural networks[J].Computer Engineering, 2019, 45(4):169-174, 180.(in Chinese)
[4] HAZARIKA D, PORIA S, ZADEH A, et al.Conversational memory network for emotion recognition in dyadic dialogue videos[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics.[S.l.]:ACL, 2018:2122-2132.
[5] MICHAEL W M, DACHER K.How emotions work:the social functions of emotional expression in negotiations[J].Research in Organizational Behavior, 2000, 22:1-50.
[6] LIU F, MAITLIS S.Emotional dynamics and strategizing processes:a study of strategic conversations in top team meetings[J].Journal of Management Studies, 2014, 51(2):202-234.
[7] PORIA S, CHATURVEDI I, CAMBRIA E, et al.Convolutional MKL based multimodal emotion recognition and sentiment analysis[C]//Proceedings of 2016 IEEE International Conference on Data Mining.Washington D.C., USA:IEEE Press, 2016:439-448.
[8] PORIA S, CAMBRIA E, HAZARIKA D, et al.Context-dependent sentiment analysis in user-generated videos[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.[S.l.]:ACL, 2017:873-883.
[9] HAZARIKA D, PORIA S, MIHALCEA R, et al.ICON:interactive conversational memory network for multimodal emotion detection[C]//Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing.Washington D.C., USA:IEEE Press, 2018:2594-2604.
[10] HAZARIKA D, PORIA S, ZIMMERMANN R, et al.Emotion recognition in conversations with transfer learning from generative conversation modeling[EB/OL].[2020-11-02].https://arxiv.org/pdf/1910.04980v1.pdf.
[11] MAJUMDER N, PORIA S, HAZARIKA D, et al.DialogueRNN:an attentive RNN for emotion detection in conversations[C]//Proceedings of AAAI Conference on Artificial Intelligence.[S.l.]:AAAI Press, 2019:6818-6825.
[12] JIN X, YU J, DING Z, et al.Hierarchical multimodal transformer with localness and speaker aware attention for emotion recognition in conversations[C]//Proceedings of CCF International Conference on Natural Language Processing and Chinese Computing.Washington D.C., USA:IEEE Press, 2020:41-53.
[13] 高玮军, 杨杰, 张春霞, 等.基于AT-DPCNN模型的情感分析研究[J].计算机工程, 2020, 46(11):53-60. GAO W J, YANG J, ZHANG C X, et al.Sentiment analysis reserach based on AT-DPCNN model[J].Computer Engineering, 2020, 46(11):53-60.(in Chinese)
[14] PORIA S, MAJUMDER N, MIHALCEA R, et al.Emotion recognition in conversation:research challenges, datasets, and recent advances[J].IEEE Access, 2019, 7:100943-100953.
[15] ZHANG D, WU L Q, SUN C L, et al.Modeling both context-and speaker-sensitive dependence for emotion detection in multi-speaker conversations[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence.Washington D.C., USA:IEEE Press, 2019:5415-5421.
[16] ZHONG P X, WANG D, MIAO C Y.Knowledge-enriched transformer for emotion detection in textual conversations[C]//Proceedings of Conference on Empirical Methods in Natural Language Processing.Washington D.C., USA:IEEE Press, 2019:165-177.
[17] XU X, RUAN Z, YANG L.Facial expression recognition based on graph neural network[C]//Proceedings of 2020 IEEE International Conference on Image, Vision and Computing.Washington D.C., USA:IEEE Press, 2020:211-214.
[18] WANG Z, TONG Y, HENG X.Phase-locking value based graph convolutional neural networks for emotion recognition[J].IEEE Access, 2019, 7:93711-93722.
[19] SONG T, ZHENG W, SONG P, et al.EEG emotion recognition using dynamical graph convolutional neural networks[J].IEEE Transactions on Affective Computing, 2020, 11(3):532-541.
[20] LO L, XIE H X, SHUAI H H, et al.MER-GCN:micro expression recognition based on relation modeling with graph convolutional network[EB/OL].[2020-11-02].https://arxiv.org/pdf/2004.08915.pdf.
[21] GHOSAL D, MAJUMDER N, PORIA S, et al.DialogueGCN:a graph convolutional neural network for emotion recognition in conversation[EB/OL].[2020-11-02].https://aclanthology.org/D19-1015.pdf.
[22] KIPF T N, WELLING M.Semi-supervised classification with graph convolutional networks[EB/OL].[2020-11-02].https://arxiv.org/pdf/1609.02907.pdf.
[23] SCHLICHTKRULL M, KIPF T N, BLOEM P, et al.Modeling relational data with graph convolutional networks[C]//Proceedings of European Semantic Web Conference.Berlin, Germany:Springer, 2018:593-607.
[24] CARLOS B, MURTAZA B, CHI-CHUN L, et al.IEMOCAP:interactive emotional dyadic motion capture database[J].Language Resources and Evaluation, 2008, 42(4):335-359.
[25] KINGMA D P, JIMMY B.Adam:a method for stochastic optimization[EB/OL].[2020-11-02].https://arxiv.org/pdf/1412.6980.pdf.
[26] SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al.Dropout:a simple way to prevent neural networks from overfitting[J].The Journal of Machine Learning Research, 2014, 15(1):1929-1958.
[27] HOCHREITER S, SCHMIDHUBER J.Long short-term memory[J].Neural Computation, 1997, 9(8):1735-1780.
[28] CHUNG J, GULCEHRE C, CHO K H, et al.Empirical evaluation of gated recurrent neural networks on sequence modeling[EB/OL].[2020-11-02].https://arxiv.org/pdf/1412.3555v1.pdf.

Please choose a citation manager

Content to export