Abstract: Visual and acoustic modalities provide important nonverbal cues for multimodal sentiment analysis, but they are also easily affected by environmental noise, individual expression differences, and modality-quality variations. When unreliable nonverbal information is directly introduced into multimodal fusion, it may cause unstable cross-modal interactions and biased sentiment prediction. To address this problem, this paper proposes a visual-acoustic residual compensation method for multimodal sentiment analysis. Instead of redesigning the whole multimodal fusion framework, the proposed method constructs a controlled residual compensation path on top of a language-focused multimodal fusion model, aiming to calibrate the remaining prediction bias of the base multimodal prediction. Specifically, textual, visual, and acoustic modalities are first encoded to generate a base multimodal prediction through a language-focused fusion structure. Then, the language-focused visual and acoustic representations are concatenated and fed into a residual compensation branch composed of fully connected mappings, nonlinear activation, dropout, shortcut connection, and a scalar output layer. This branch produces a visual-acoustic compensation value for output-level incremental calibration. To avoid excessive correction caused by auxiliary modalities, a forward compensation weight, a residual clipping threshold, a compensation amplitude regularization term, and a residual supervision objective are introduced. In this way, the compensation branch is encouraged to learn the remaining bias between the base prediction and the ground-truth sentiment label, rather than directly fitting the complete sentiment target. Considering that the stability of base prediction and residual patterns may vary under different data conditions, two residual-training paths are designed. For CMU-MOSEI, the model adopts a Residual+Stage path, in which the base prediction branch is trained first, the residual compensation branch is then optimized, and the whole model is finally fine-tuned jointly. For CMU-MOSI, the model adopts a Residual-only path, where the base prediction branch and the residual compensation branch are optimized end-to-end. To reduce the confounding factors caused by cross-dataset comparison, controlled down-sampling experiments with different training scales are further conducted on CMU-MOSEI while keeping the validation set, test set, modality features, and evaluation scripts unchanged. Experimental results show that the proposed method outperforms the DLF baseline on all six evaluation metrics on CMU-MOSEI and CMU-MOSI. On CMU-MOSEI, the proposed method achieves 55.26% Acc-7, 57.04% Acc-5, 86.91% Acc-2, and 86.88% F1, improving over DLF by 1.36, 1.34, 1.49, and 1.61 percentage points, respectively. The correlation coefficient increases from 0.764 to 0.780, and the mean absolute error decreases from 0.536 to 0.530. On CMU-MOSI, the proposed method achieves 48.19% Acc-7, 54.50% Acc-5, 86.67% Acc-2, and 87.70% F1, improving over DLF by 1.11, 2.17, 1.61, and 2.66 percentage points, respectively. The correlation coefficient increases from 0.781 to 0.800, and the mean absolute error decreases from 0.731 to 0.721. Ablation results show that different residual-training paths are suitable for different data conditions. On CMU-MOSEI, Residual+Stage achieves the best overall performance, while on CMU-MOSI, Residual-only performs better. The controlled training-scale experiments on CMU-MOSEI further show that Residual-only is more stable when only 10% or 25% of the training data is used, whereas Residual+Stage gradually becomes more effective as the training ratio increases to 50%, 75%, and 100%. This indicates that training sample size affects the stability of base prediction error patterns and the adaptability of residual-training paths. The compensation-value analysis shows that the average absolute compensation values on CMU-MOSEI and CMU-MOSI are only 0.018 and 0.021, respectively, and the proportions of samples close to the compensation boundary are 2.6% and 3.1%. These results indicate that the compensation branch does not largely rewrite the base prediction, but performs controlled calibration within a small range. The computational cost analysis further shows that the residual compensation branch introduces only about 2.03×104 additional parameters, increasing the total parameter size from 111.23M to 111.25M, with only slight increases in training and inference time. Overall, the proposed method provides an effective output-level calibration strategy for controlled use of visual and acoustic modalities, and improves multimodal sentiment prediction with limited additional computational cost.

摘要： 针对多模态情感分析中视觉、音频等非语言模态容易受到采集环境、个体表达差异和模态质量波动影响，从而在融合过程中引入噪声并造成预测偏差的问题，本文提出一种视听残差补偿多模态情感分析方法。该方法的研究目的并不是重新构建完整的多模态融合框架，而是在已有语言聚焦多模态融合模型的基础预测结果之外，增加一条受控的视觉-音频残差补偿路径，使非语言模态能够在参与基础融合预测的同时，对基础预测中尚未被充分刻画的剩余偏差进行输出级校准。具体而言，模型首先对文本、视觉和音频三种模态进行编码，并利用语言聚焦的多模态融合结构生成基础多模态预测结果；随后，将语言聚焦后的视觉增强表示和音频增强表示进行拼接，输入由全连接映射、非线性激活、Dropout、短接连接和标量输出层组成的残差补偿分支，得到视觉-音频补偿项。为避免补偿分支对基础预测结果产生过量修正，本文进一步引入前向补偿权重、残差截断阈值、补偿幅值正则项和残差监督目标，使补偿分支学习基础预测结果与真实情感标签之间的剩余偏差，而不是重新拟合完整情感标签。考虑到不同数据条件下基础预测稳定性和残差模式充分性可能存在差异，本文分别设计端到端残差补偿训练路径和残差补偿结合分阶段训练路径。在 CMU-MOSEI 数据集上，模型采用先训练基础预测分支、再训练残差补偿分支、最后联合微调的 Residual+Stage 路径；在 CMU-MOSI 数据集上，模型采用基础预测分支与残差补偿分支同步更新的 Residual-only 路径。为避免仅依赖跨数据集比较带来的混杂因素，本文还在 CMU-MOSEI 上构造不同训练规模的下采样受控实验，通过保持验证集、测试集、模态特征配置和评价脚本不变，观察训练样本规模变化对残差训练路径适配性的影响。实验结果表明，本文方法在 CMU-MOSEI 和 CMU-MOSI 两个公开数据集的六项评价指标上均优于 DLF 基线模型。在 CMU-MOSEI 上，本文方法的 Acc-7、Acc-5、Acc-2 和 F1 分别达到 55.26%、57.04%、86.91% 和 86.88%，相较于 DLF 分别提升 1.36、1.34、1.49 和 1.61个百分点；相关系数由 0.764 提高至 0.780，平均绝对误差由 0.536 降低至 0.530。在 CMU-MOSI 上，本文方法的 Acc-7、Acc-5、Acc-2 和 F1 分别达到 48.19%、54.50%、86.67% 和 87.70%，相较于 DLF 分别提升 1.11、2.17、1.61 和 2.66个百分点；相关系数由 0.781 提高至 0.800，平均绝对误差由 0.731 降低至 0.721。结构消融实验进一步表明，在 CMU-MOSEI 上，Residual+Stage 路径取得最佳综合性能，F1、Corr 和 MAE 分别为 86.88%、0.780 和 0.530；而在 CMU-MOSI 上，Residual-only 路径表现更优，F1、Corr 和 MAE 分别为 87.70%、0.800 和 0.721，说明残差补偿分支的效果与训练路径组织方式密切相关。损失项消融实验显示，加入残差监督损失、补偿幅值正则项和样本加权机制后，模型性能逐步提升，说明这些约束能够改善补偿方向、限制过量修正并增强残差学习效果。训练规模受控实验显示，当 CMU-MOSEI 训练比例为 10% 和 25% 时，Residual-only 表现更稳定；当训练比例增加到 50%、75% 和 100% 时，Residual+Stage 逐渐取得优势，表明训练样本规模变化会影响基础预测误差模式的稳定性和补偿分支训练路径的适配效果。补偿量统计结果进一步显示，CMU-MOSEI 和 CMU-MOSI 上实际补偿量的平均绝对值分别为 0.018 和 0.021，接近补偿边界的样本比例分别仅为 2.6% 和 3.1%，说明补偿分支并未对基础预测进行大幅度改写，而是在较小范围内进行受控校准；两个数据集上的补偿方向一致率分别为 63.4% 和 65.7%，有效校准率分别为 58.9% 和 60.8%，进一步说明补偿分支能够在多数样本上朝降低基础预测误差的方向修正。计算开销分析表明，残差补偿分支仅增加约 2.03×104个参数，模型参数量由 111.23M 增加到 111.25M；CMU-MOSEI 和 CMU-MOSI 上单轮训练时间分别增加约 2.5% 和 3.1%，单样本推理时间均增加约 2.2%，说明该补偿机制在提升预测性能的同时并未显著增加计算负担。综合实验结果表明，视听残差补偿能够以较小的参数和时间开销改善基础多模态预测结果，残差监督与幅值约束有助于降低非语言模态导致过量修正的风险，而不同训练路径在不同数据条件下具有不同适配性。该方法为多模态情感分析中辅助模态的受控利用和输出级偏差校准提供了一种可行思路。