Multi-modal Sentiment Analysis Based on Private Feature Learning and Contrastive Learning

doi:10.19678/j.issn.1000-3428.0252268

Abstract

Abstract: In multi-modal sentiment analysis tasks, traditional methods rely on the direct fusion of multi-modal information and private features unique to each modality are often overlooked by cross-modal interactions. This may result in insufficient accuracy and robustness when the model handles complex emotional expressions. This is particularly critical in smart education scenarios, where teachers need to precisely assess students' learning states and emotional fluctuations through their voices, facial expressions, and textual feedback. Therefore, improving the accuracy of multi-modal sentiment analysis is of great significance for personalized teaching and classroom interactions. To address this issue, a sentiment analysis model that combines private feature learning and contrastive learning is proposed. First, to fully exploit private features, the model compares shared features with the original text, audio, and visual features based on similarity, thereby identifying private features that are neglected during cross-modal interactions. The model enhances its expressive capabilities by fusing private and shared features. Second, a Modality-Agnostic Contrastive Loss (MACL) is proposed, which effectively leverages sentiment information in multi-modal data through contrastive learning on multi-modal fused features, reducing inter-modal discrepancies and thereby obtaining unified sentiment representations. The experimental results show that on the CMU-MOSI and CMU-MOSEI datasets, the proposed model achieves F1 values of 85.98% and 85.95% and binary classification accuracies of 86.01% and 85.97%, respectively, significantly outperforming the second-best models, which validates the effectiveness of the proposed model.

Key words: multi-modal sentiment analysis, private feature learning, contrastive learning, feature fusion, sentiment prediction

摘要： 在多模态情感分析任务中，传统方法依赖于直接融合多模态信息，而每个模态特有的私有特征往往被跨模态交互所忽略，这可能导致模型在处理复杂情感表达时准确性和鲁棒性不足，特别是在智慧教育场景中，教师需要通过学生的语音、表情和文本反馈来精准判断其学习状态和情绪波动，因此，提升多模态情感分析的精度对于个性化教学和课堂交互具有重要意义。为了解决这一问题，提出一种结合私有特征学习和对比学习的情感分析模型。首先，为了充分利用私有特征，该模型将共享特征与原始的文本、音频、视觉特征进行相似性比较，从而识别在跨模态交互中被忽视的私有特征，通过融合私有特征和共享特征来增强模型的表达能力；其次，提出一种模态无关对比损失（MACL），通过对多模态融合特征进行对比学习，有效利用多模态数据中的情感信息，减少模态间的差距，进而获得统一的情感表示。实验结果表明，在CMU-MOSI和CMU-MOSEI数据集上，该模型的F1值分别达到85.98%和85.95%，二分类准确率分别达到86.01%和85.97%，显著高于次优模型，验证了该模型的有效性。

关键词: 多模态情感分析, 私有特征学习, 对比学习, 特征融合, 情感预测

CLC Number:

TP391

FENG Guang, XIANG Feng, HUANG Rongcan, ZHOU Yuanhua, ZHENG Runting, YANG Yanru, LIU Tianxiang, LI Weichen. Multi-modal Sentiment Analysis Based on Private Feature Learning and Contrastive Learning[J]. Computer Engineering, 2026, 52(11): 331-342.

冯广, 项峰, 黄荣灿, 周垣桦, 郑润庭, 杨燕茹, 刘天翔, 李伟辰. 基于私有特征学习与对比学习的多模态情感分析[J]. 计算机工程, 2026, 52(11): 331-342.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0252268

https://www.ecice06.com/EN/Y2026/V52/I11/331

References

[1] 冯广, 江家懿, 罗时强, 等. 基于话语间时序多模态数据的情绪分析方法[J]. 计算机系统应用, 2022, 31(5): 195-202.FENG G, JIANG J Y, LUO S Q, et al. Sentiment analysis method based on temporal multimodal data between utterances[J]. Computer Systems & Applications, 2022, 31(5): 195-202. (in Chinese)
[2] 冯广, 鲍龙. 基于红外可见光融合的复杂环境下人脸识别方法[J]. 广东工业大学学报, 2024, 41(3): 62-70, 109.FENG G, BAO L. Face recognition method in complex environment based on infrared visible fusion[J]. Journal of Guangdong University of Technology, 2024, 41(3): 62-70, 109. (in Chinese)
[3] 吴亚迪, 陈平华. 基于用户长短期偏好和音乐情感注意力的音乐推荐模型[J]. 广东工业大学学报, 2023, 40(4): 37-44.WU Y D, CHEN P H. A music recommendation model based on users’long and short term preferences and music emotional attention[J]. Journal of Guangdong University of Technology, 2023, 40(4): 37-44. (in Chinese)
[4] MAI S J, HU H F, XING S L. Modality to modality translation: an adversarial representation learning and graph fusion network for multimodal fusion[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(1): 164-172.
[5] PANDEYA Y R, LEE J. Deep learning-based late fusion of multimodal information for emotion classification of music video[J]. Multimedia Tools and Applications, 2021, 80(2): 2887-2905.
[6] HAN W, CHEN H, GELBUKH A, et al. Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis[C]//Proceedings of the 2021 International Conference on Multimodal Interaction. Washington D.C.,USA:IEEE Press,2021: 6-15.
[7] 袁萍萍. 基于对比学习的多模态特征融合情感分析算法研究[D]. 南昌: 南昌大学, 2023.YUAN P P. Research on multimodal feature fusion sentiment analysis algorithm based on contrast learning[D]. Nanchang: Nanchang University, 2023. (in Chinese)
[8] HAN W, CHEN H, PORIA S. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. [S.l.]:ACL,2021: 9180-9192.
[9] LIN Z J, LIANG B, LONG Y F, et al. Modeling intra- and inter-modal relations: hierarchical graph contrastive learning for multimodal sentiment analysis[C]//Proceedings of the 29th International Conference on Computational Linguistics. [S.l.]:ACL,2022: 7124-7135.
[10] MA F P, ZHANG Y Y, SUN X Y. Multimodal sentiment analysis with preferential fusion and distance-aware contrastive learning[C]//Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). Washington D.C.,USA:IEEE Press,2023: 1367-1372.
[11] ZADEH A, ZELLERS R, PINCUS E, et al. Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages[J]. IEEE Intelligent Systems, 2016, 31(6): 82-88.
[12] BAGHER Z A, LIANG P P, PORIA S, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. [S.l.]:ACL,2018: 2236-2246.
[13] ZADEH A, CHEN M H, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. [S.l.]:ACL,2017: 1103-1114.
[14] LIU Z, SHEN Y, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. [S.l.]:ACL,2018: 2247-2256.
[15] ZADEH A, LIANG P P, MAZUMDER N, et al. Memory fusion network for multi-view sequential learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S.l.]: AAAI Press, 2018: 15-22.
[16] HAZARIKA D, ZIMMERMANN R, PORIA S. MISA: modality-invariant and-specific representations for multimodal sentiment analysis[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York,USA:ACM Press,2020: 1122-1131.
[17] YU W M, XU H, YUAN Z Q, et al. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S.l.]: AAAI Press, 2021: 10790-10797.
[18] TSAI Y H, BAI S J, LIANG P P, et al. Multimodal transformer for unaligned multimodal language sequences[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. [S.l.]:ACL,2019: 6558-6569.
[19] WANG D, LIU S, WANG Q, et al. Cross-modal enhancement network for multimodal sentiment analysis[J]. IEEE Transactions on Multimedia, 2023, 25: 4909-4921.
[20] RAHMAN W, HASAN M K, LEE S, et al. Integrating multimodal information in large pretrained transformers[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. [S.l.]:ACL,2020: 2359-2369.
[21] WANG L, PENG J J, ZHENG C Z, et al. A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning[J]. Information Processing & Management, 2024, 61(3): 103675.
[22] HUANG J H, ZHOU J, TANG Z C, et al. TMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis[J]. Knowledge-Based Systems, 2024, 285: 111346.
[23] MAI S J, SUN Y, ZENG Y, et al. Excavating multimodal correlation for representation learning[J]. Information Fusion, 2023, 91: 542-555.
[24] LAI S N, LI J K, GUO G N, et al. Shared and private information learning in multimodal sentiment analysis with deep modal alignment and self-supervised multi-task learning[C]//Proceedings of the International Joint Conference on Neural Networks (IJCNN). Washington D.C.,USA:IEEE Press,2024: 1-8.
[25] JAISWAL A, BABU A R, ZADEH M Z, et al. A survey on contrastive self-supervised learning[J]. Technologies, 2021, 9(1): 2.
[26] CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[EB/OL].[2025-01-05]. https://arxiv.org/abs/2002.05709.
[27] GAO T Y, YAO X C, CHEN D Q. SimCSE: simple contrastive learning of sentence embeddings[EB/OL].[2025-01-05]. https://arxiv.org/abs/2104.08821.
[28] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[EB/OL].[2025-01-05]. https://arxiv.org/abs/2103.00020.
[29] YANG J D, YU Y K, NIU D, et al. ConFEDE: contrastive feature decomposition for multimodal sentiment analysis[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. [S.l.]:ACL,2023: 7617-7630.
[30] MAI S J, ZENG Y, ZHENG S J, et al. Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis[J]. IEEE Transactions on Affective Computing, 2023, 14(3): 2276-2289.
[31] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL].[2025-01-05]. https://arxiv.org/abs/1810.04805.
[32] HOCHREITER S, SCHMIDHUBER J. Long short-term memory[J]. Neural Computation, 1997, 9(8): 1735-1780.
[33] HU J, SHEN L, SUN G. Squeeze-and-Excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C.,USA:IEEE Press,2018: 7132-7141.
[34] WANG F, JIANG M Q, QIAN C, et al. Residual attention network for image classification[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C.,USA:IEEE Press,2017: 6450-6458.
[35] WU T, PENG J J, ZHANG W Q, et al. Video sentiment analysis with bimodal information-augmented multi-head attention[J]. Knowledge-Based Systems, 2022, 235: 107676.
[36] LIN R H, HU H F. Multimodal contrastive learning via uni-modal coding and cross-modal prediction for multimodal sentiment analysis[C]//Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2022. [S.l.]:ACL,2022: 511-523.
[37] LIU S, LUO Z, FU W N. Fcdnet: fuzzy cognition-based dynamic fusion network for multimodal sentiment analysis[J]. IEEE Transactions on Fuzzy Systems, 2025, 33(1): 3-14.

Please choose a citation manager

Content to export