Abstract:

To prevent important information containing emotional cues from being obscured by irrelevant information in discourse and to achieve multi-modal information interaction, a multi-modal emotion recognition model based on dynamic convolution and residual gating is proposed by mining advanced local features and designing effective interaction fusion strategies. Low-level features, high-level local features, and contextual dependencies from text, audio, and images are extracted. While using cross modal dynamic convolution to model inter-modal and intra-modal interactions, interactions are simulated between long sequences in time domain, and interaction features of different modalities are captured. A residual gated fusion method that fuses different modal interaction representations automatically learns the impact weight of each interaction feature on the final output, and inputs the multi-modal fusion feature into the classifier for emotion prediction. The experimental results show that this model prevents important information regarding emotional cues from being obscured by irrelevant information in multi-modal data. The accuracy of sentiment classification is 83.5% and 83.9% on the CMU-MOSEI and IEMOCAP datasets, respectively. The model outperforms benchmark models such as Multi-modal Transformer(MulT) and Multi-Fusion Residual Memory(MFRM).

Key words: natural language processing, information interaction, multi-modal emotion recognition, dynamic convolution, gating mechanism

摘要：

为了防止一段话语中含有情感色彩的重要信息被无关信息淹没并实现多模态信息交互，通过挖掘高级局部特征以及设计有效的交互融合策略，提出一种基于动态卷积与残差门控的多模态情感识别模型。提取文本、音频和图像中的低级特征、高级局部特征以及上下文依赖关系，同时使用跨模态动态卷积对模态间和模态内交互信息进行建模，模拟长序列时域间的相互作用，捕捉不同模态的交互特征。设计一种残差门控融合方法来融合不同模态交互表征，自动学习每组交互表征对最终情感识别的影响权重，并将多模态融合特征输入分类器进行情感预测。在CMU-MOSEI和IEMOCAP数据集上的实验结果表明，该模型能够避免多模态中含有情感色彩的重要信息被无关信息淹没，情感分类准确率分别达到83.5%和83.9%，性能优于MulT、MFRM等基准模型。

关键词: 自然语言处理, 信息交互, 多模态情感识别, 动态卷积, 门控机制