融合混合编码与模糊建模的多模态对话情感识别模型

doi:10.19678/j.issn.1000-3428.0260120

摘要/Abstract

摘要： 多模态对话情感识别(ERC)通过融合语言、声学和视觉等多源信息,实现对话情绪的自动识别,从而增强人机交互的自然性与情感理解。然而,现有方法在建模情感的多层上下文依赖方面仍存在不足,模态融合易引入冗余或噪声,且难以刻画情感的不确定性,限制复杂情绪识别。针对上述问题,提出一种融合混合编码与模糊建模的多模态ERC模型。该模型通过混合编码模块同时建模情感的全局对话上下文与局部依赖关系,从而增强情感时序特征的表达能力,并在此基础上引入分层门控融合机制,对不同层次和不同模态特征进行动态加权融合,以有效抑制冗余信息与噪声干扰。在情感分类阶段,采用线性等间距初始化的模糊神经网络,通过模糊隶属函数对情感类别边界进行建模,以刻画情绪表达中的不确定性与模糊性。实验结果显示,该模型在IEMOCAP、MELD和CMU-MOSEI 3个数据集上的各项指标均优于基线模型,在IEMOCAP上准确率为72.67%,在MELD上准确率为67.37%,在CMU-MOSEI七分类与二分类准确率上分别为54.96%和86.78%,验证了所提模型在多模态情感分析中的有效性。

关键词: 多模态, 对话情感识别, 上下文建模, 模态融合, 模糊神经网络

Abstract: Multimodal Emotion Recognition in Conversations (ERC) integrates language, acoustic, and visual information to automatically identify emotions in dialogues, thereby enhancing naturalness and emotional understanding in human-computer interactions. However, existing methods have limitations in modeling the multilayer contextual dependencies of emotions. Multimodal feature fusion often introduces redundant information and noise, making it difficult to capture the uncertainty of emotions, thereby limiting the recognition of complex emotions. To address these issues, this study proposes a multimodal ERC model that combines hybrid encoding and fuzzy modeling. The model uses a hybrid encoding module to simultaneously capture the global dialogue context and local dependencies of emotions simultaneously, thereby enhancing the representational capability of emotional temporal features. Accordingly, a hierarchical gated fusion mechanism is introduced to perform dynamic weighted fusion on features from different levels and modalities, effectively suppressing redundant information and noise. In the emotion classification stage, a fuzzy neural network with linear equally spaced initialization models the boundaries of emotion categories using fuzzy membership functions to capture the uncertainty and fuzziness in emotional expressions. The experimental results show that the proposed model outperforms the baseline models for all metrics across the IEMOCAP, MELD, and CMU-MOSEI datasets. It achieves an accuracy of 72.67% for IEMOCAP, 67.37% for MELD, 54.96% for seven-class classification accuracy, and 86.78% for two-class classification accuracy on CMU-MOSEI, validating the effectiveness of the proposed model in multimodal emotion recognition.

Key words: multimodal, Emotion Recognition in Conversations (ERC), contextual modeling, modal fusion, fuzzy neural network

中图分类号:

TP391

钟杭, 张清华, 罗南方, 郭芮利. 融合混合编码与模糊建模的多模态对话情感识别模型[J]. 计算机工程, 2026, 52(10): 317-327.

ZHONG Hang, ZHANG Qinghua, LUO Nanfang, GUO Ruili. Multimodal Emotion Recognition in Conversations Model Fused with Hybrid Encoding and Fuzzy Modeling[J]. Computer Engineering, 2026, 52(10): 317-327.

https://www.ecice06.com/CN/Y2026/V52/I10/317

参考文献

[1] RASHKIN H, SMITH E M, LI M, et al. Towards empathetic open-domain conversation models: a new benchmark and dataset[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Philadelphia, USA: Association for Computational Linguistics, 2019: 5370-5381.
[2] 赵妍妍, 陆鑫, 赵伟翔, 等. 情感对话技术综述[J]. 软件学报, 2024, 35(3): 1377-1402. ZHAO Y Y, LU X, ZHAO W X, et al. Survey on emotional dialogue techniques[J]. Journal of Software, 2024, 35(3): 1377-1402. (in Chinese)
[3] 陈晓婷, 李实. 对话情绪识别综述[J]. 计算机工程与应用, 2023, 59(3): 33-48. CHEN X T, LI S. Survey on emotion recognition in conversation[J]. Computer Engineering and Applications, 2023, 59(3): 33-48. (in Chinese)
[4] JORDAN M I. Serial order: a parallel distributed processing approach[M]. [S. l.]: North-Holland, 1997.
[5] GORI M, MONFARDINI G, SCARSELLI F. A new model for learning in graph domains[C]//Proceedings of the 2005 IEEE International Joint Conference on Neural Networks. Washington D.C., USA: IEEE Press, 2005: 729-734.
[6] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, USA: Curran Associates Inc., 2017: 6000-6010.
[7] BUSSO C, BULUT M, LEE C C, et al. IEMOCAP: interactive emotional dyadic motion capture database[J]. Language Resources and Evaluation, 2008, 42(4): 335-359.
[8] PORIA S, HAZARIKA D, MAJUMDER N, et al. MELD: a multimodal multi-party dataset for emotion recognition in conversations[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Philadelphia, USA: Association for Computational Linguistics, 2019: 527-536.
[9] ZADEH A B, LIANG P P, PORIA S, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Philadelphia, USA: Association for Computational Linguistics, 2018: 2236-2246.
[10] 化青远, 彭涛, 崔海, 等. 基于知识图谱中路径推理的多轮对话模型[J]. 吉林大学学报(理学版), 2025, 63(1): 76-82. HUA Q Y, PENG T, CUI H, et al. Multi round conversational model based on path reasoning in knowledge graph[J]. Journal of Jilin University (Science Edition), 2025, 63(1): 76-82. (in Chinese)
[11] CHO K, VAN MERRIËNBOER B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Philadelphia, USA: Association for Computational Linguistics, 2014: 1724-1734.
[12] GRAVES A. Long short-term memory[M]//GRAVES A. Supervised sequence labelling with recurrent neural networks. Berlin, Germany: Springer, 2012: 37-45.
[13] PORIA S, CAMBRIA E, HAZARIKA D, et al. Context-dependent sentiment analysis in user-generated videos[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Philadelphia, USA: Association for Computational Linguistics, 2017: 873-883.
[14] MAJUMDER N, PORIA S, HAZARIKA D, et al. DialogueRNN: an attentive RNN for emotion detection in conversations[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2019: 6818-6825.
[15] SHENOY A, SARDANA A. Multilogue-Net: a context-aware RNN for multi-modal emotion detection and sentiment analysis in conversation[C]//Proceedings of the 2nd Grand-Challenge and Workshop on Multimodal Language (Challenge-HML). Philadelphia, USA: Association for Computational Linguistics, 2020: 19-28.
[16] HU D, WEI L W, HUAI X Y. DialogueCRN: contextual reasoning networks for emotion recognition in conversations[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Philadelphia, USA: Association for Computational Linguistics, 2021: 7042-7052.
[17] GHOSAL D, MAJUMDER N, PORIA S, et al. DialogueGCN: a graph convolutional neural network for emotion recognition in conversation[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Philadelphia, USA: Association for Computational Linguistics, 2019: 154-164.
[18] FENG J W, FAN X Y. Cross-modal context fusion and adaptive graph convolutional network for multimodal conversational emotion recognition[C]//Proceedings of the International Joint Conference on Neural Networks (IJCNN). Washington D.C., USA: IEEE Press, 2025: 1-8.
[19] LI J Y, JI D H, LI F, et al. HiTrans: a transformer-based context- and speaker-sensitive model for emotion detection in conversations[C]//Proceedings of the 28th International Conference on Computational Linguistics. Philadelphia, USA: Association for Computational Linguistics, 2020: 4190-4200.
[20] FU Y M, WU J J, WANG Z J, et al. LaERC-S: improving LLM-based emotion recognition in conversation with speaker characteristics[C]//Proceedings of the 31st International Conference on Computational Linguistics. Philadelphia, USA: Association for Computational Linguistics, 2025: 6748-6761.
[21] ZADEH L A. Fuzzy sets[J]. Information and Control, 1965, 8(3): 338-353.
[22] NGUYEN T L, KAVURI S, LEE M. A multimodal convolutional neuro-fuzzy network for emotion understanding of movie clips[J]. Neural Networks, 2019, 118: 208-219.
[23] CHATURVEDI I, SATAPATHY R, CAVALLARI S, et al. Fuzzy commonsense reasoning for multimodal sentiment analysis[J]. Pattern Recognition Letters, 2019, 125: 264-270.
[24] JIANG D Z, LIU H, WEI R G, et al. CSAT-FTCN: a fuzzy-oriented model with contextual self-attention network for multimodal emotion recognition[J]. Cognitive Computation, 2023, 15(3): 1082-1091.
[25] NGUYEN N M, NGUYEN T M, NGUYEN T T, et al. Enhancing multimodal emotion recognition with dynamic fuzzy membership and attention fusion[J]. Engineering Applications of Artificial Intelligence, 2026, 165: 113396.
[26] HU D, HOU X L, WEI L W, et al. MM-DFN: multimodal dynamic fusion network for emotion recognition in conversations[C]//Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Washington D.C., USA: IEEE Press, 2022: 7037-7041.
[27] MENG T, ZHANG F C, SHOU Y T, et al. Masked graph learning with recurrent alignment for multimodal emotion recognition in conversation[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024, 32: 4298-4312.
[28] DAI Y J, LI J X, LI Y J, et al. Multi-modal graph context extraction and consensus-aware learning for emotion recognition in conversation[J]. Knowledge-Based Systems, 2024, 298: 111954.
[29] LI X R, XU X J, QIAO J Q, et al. Long-short distance graph neural networks and improved curriculum learning for emotion recognition in conversation[C]//Proceedings of the ECAI 2025. [S. l.]: IOS Press, 2025: 1-12.
[30] FU C Z, QIAN F K, SU K F, et al. HiMul-LGG: a hierarchical decision fusion-based local-global graph neural network for multimodal emotion recognition in conversation[J]. Neural Networks, 2025, 181: 106764.
[31] XIE X T, MA T H, JIA L, et al. Hypergraph-based multimodal adaptive fusion for emotion recognition in conversation[J]. The Journal of Supercomputing, 2025, 81(10): 1128.
[32] ZADEH A, CHEN M H, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Philadelphia, USA: Association for Computational Linguistics, 2017: 1103-1114.
[33] LIU Z, SHEN Y, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Philadelphia, USA: Association for Computational Linguistics, 2018: 2247-2256.
[34] HUANG J H, ZHOU J, TANG Z C, et al. TMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis[J]. Knowledge-Based Systems, 2024, 285: 111346.
[35] ZHU L N, ZHAO H Y, ZHU Z C, et al. Multimodal sentiment analysis with unimodal label generation and modality decomposition[J]. Information Fusion, 2025, 116: 102787.
[36] CHEN J W, SONG S X, TAN Y M, et al. TEMSA: text enhanced modal representation learning for multimodal sentiment analysis[J]. Computer Vision and Image Understanding, 2025, 258: 104391.

选择文件类型/文献管理软件名称

选择包含的内容