Abstract: To address significant heterogeneity among modality representations, redundant noise introduced by non-textual modalities such as vision and audio, and the difficulty of dynamically modeling sample-specific modality contributions in multimodal sentiment analysis, this paper proposes a Text-Guided and Dual-Gated Adaptive Graph Fusion model, namely TGAGF. In video-based sentiment analysis, textual, visual, and acoustic modalities provide sentiment cues from semantic content, facial expressions, and acoustic variations, respectively. However, these three modalities differ significantly in representation formats, information density, and noise distributions. The textual modality usually has strong semantic representation capability and can more directly reflect the speaker’s opinions and sentiment orientation. Visual and acoustic modalities can provide complementary nonverbal information, such as facial expressions, head poses, intonation, and speaking rate, but they are also susceptible to individual differences, environmental interference, and acquisition quality. If multimodal features are directly fused, weakly relevant information and redundant noise from non-textual modalities may be introduced into the joint representation and further propagated or amplified during cross-modal interaction, thereby weakening the model’s discriminative ability for sentiment polarity, fine-grained sentiment levels, and continuous sentiment intensity. To this end, TGAGF takes the textual modality as semantic guidance to enhance the representation capability of effective sentiment cues in visual and acoustic modalities during cross-modal fusion, while suppressing the influence of redundant noise on sentiment prediction. First, the model performs feature extraction and dimensional alignment on textual, visual, and acoustic modalities, mapping features from different modalities into a unified semantic space to provide structurally consistent input representations for subsequent fusion. Then, a shared-specific dual-branch disentanglement network is introduced to separate cross-modal consistent information from modality-specific information. Specifically, the shared branch extracts sentiment-related consistent semantic information across different modalities, while the modality-specific branch preserves complementary discriminative information within each modality. Meanwhile, reconstruction consistency, modality-specific consistency, metric constraint, and orthogonality constraint are jointly employed to constrain the disentangled representations from the perspectives of semantic recovery, modality-specific stability, cross-modal consistency, and subspace independence, thereby reducing semantic entanglement between shared and specific features. Second, a text-guided dual-stage gating mechanism is adopted to suppress redundant noise in visual and acoustic modalities before graph-based relational reasoning and to reweight modality representations after graph propagation. The first-stage gate pre-selects the modality information entering the graph reasoning module, reducing the diffusion of invalid information during relational propagation. The second-stage gate readjusts the contribution of each modality according to the cross-modal interaction results after graph propagation, enabling the model to dynamically allocate modality weights according to the sentiment expression characteristics of different samples. On this basis, textual, visual, and acoustic modalities are regarded as graph nodes, and a learnable adjacency matrix is used to model inter-modal dependencies. In this way, semantic correlations among different modalities are explicitly captured at the modality level, and complementary information across modalities is enhanced through graph message propagation. Finally, a cascaded multi-task prediction strategy is employed to jointly model sentiment polarity, fine-grained sentiment levels, and continuous sentiment intensity, allowing the model to simultaneously perform sentiment direction discrimination, sentiment level classification, and continuous sentiment intensity estimation. Experimental results on the CMU-MOSI and CMU-MOSEI datasets show that the proposed model achieves competitive performance on multiple evaluation metrics. Specifically, the Acc7, Acc5, Acc2, and F1 scores on CMU-MOSI are 48.4%, 56.1%, 85.3%, and 85.3%, respectively; the Acc2, F1, and MAE on CMU-MOSEI are 85.2%, 85.1%, and 0.534, respectively. Compared with various classical methods and recent representative methods, TGAGF demonstrates strong competitiveness in both sentiment classification and sentiment intensity prediction tasks. The experimental results verify the effectiveness of the proposed method in alleviating modality heterogeneity, suppressing redundant noise interference, dynamically modeling modality contributions, and improving the stability of multimodal sentiment recognition.

摘要： 针对多模态情感分析中模态表示异质性强、视觉和音频等非文本模态易引入冗余噪声，以及不同样本下模态贡献难以动态建模等问题，提出一种文本引导与双门控自适应图融合模型（Text-Guided and Dual-Gated Adaptive Graph Fusion，TGAGF）。在视频情感分析任务中，文本、视觉和音频分别从语义内容、面部表情和声学变化等方面提供情感线索，但三类模态在表示形式、信息密度和噪声分布上存在明显差异。文本模态通常具有较强的语义表达能力，能够更直接地反映说话人的观点和情感倾向；视觉和音频模态能够补充表情、姿态、语调和语速等非语言信息，但也容易受到个体差异、环境干扰和采集质量等因素的影响。若直接对多模态特征进行融合，非文本模态中的弱相关信息和冗余噪声可能被引入联合表示，并在跨模态交互过程中进一步传播或放大，从而削弱模型对情感极性、细粒度等级和连续情感强度的判别能力。为此，TGAGF以文本模态为语义引导，在跨模态融合过程中增强视觉和音频模态中有效情感线索的表征能力，并抑制冗余噪声对情感判别结果的影响。首先，模型对文本、视觉和音频三种模态进行特征提取与维度对齐，将不同模态特征映射到统一语义空间，为后续融合提供结构一致的输入表示。随后，通过共享—特有双分支解耦网络分离跨模态一致信息与模态特有信息。其中，共享分支提取与情感表达相关的一致语义信息，特有分支保留各模态自身的补充判别信息；同时结合重建一致性、特定一致性、度量约束和正交约束，从语义恢复、模态稳定性、跨模态一致性和子空间独立性等方面约束解耦表示，降低共享特征与特有特征之间的语义混叠。其次，利用文本引导双阶段门控机制在图关系推理前抑制视觉和音频模态中的冗余噪声，在图传播后重新加权各模态表示。第一阶段门控对进入图关系推理模块的模态信息进行预筛选，减少无效信息在关系传播过程中的扩散；第二阶段门控依据图传播后的跨模态交互结果重新调整各模态贡献，使模型能够根据不同样本的情感表达特点动态分配模态权重。在此基础上，将文本、视觉和音频三种模态作为图节点，通过可学习邻接矩阵建模模态间依赖关系，在模态级显式刻画不同模态之间的语义相关性，并通过图消息传播增强模态间互补信息。最后，采用级联多任务预测策略联合建模情感极性、细粒度等级和连续情感强度，使模型同时兼顾情感方向判别、情感等级区分和连续强度估计。在CMU-MOSI和CMU-MOSEI数据集上的实验结果表明，所提模型在多项评价指标上取得较优性能。其中，在CMU-MOSI上的Acc7、Acc5、Acc2和F1分别为48.4%、56.1%、85.3%和85.3%；在CMU-MOSEI上的Acc2、F1和MAE分别为85.2%、85.1%和0.534。与多种经典方法和近期代表性方法相比，TGAGF在情感分类和情感强度预测任务中均表现出较强竞争力。实验结果验证了所提方法在缓解模态异质性、抑制冗余噪声干扰、动态