Abstract: Trusted multimodal classification, which integrates heterogeneous data streams and quantifies predictive uncertainty, is of great significance in high risk domains such as medical diagnosis and autonomous driving. However, when handling multimodal conflicts, most existing methods still suffer from three core issues: mechanical noise processing leads to overconfidence when evidence is scarce; the unified fusion paradigm causes modality specific deep semantics to be lost due to “feature assimilation”; and uncertainty estimates are not utilized as a dynamic feedback signal to guide fusion decisions in a closed loop manner. To address these challenges, this paper proposes a Joint Adversarial like Learning and Dynamic Fusion (JALDF) method for trustworthy multimodal classification. The method aims to systematically improve the accuracy, noise robustness, and trustworthiness of multimodal classification through the synergistic optimization of feature alignment, semantic preservation, and adaptive fusion. In terms of technical implementation, JALDF consists of three core innovative modules. First, the cross modal class similarity module applies sigmoid sparsification and L₁ regularization to each modality’s features to select highly discriminative subsets, then extracts samples by class and uses the Wasserstein distance to measure the distribution discrepancy of the same class across different modalities. By minimizing this distance, the network is forced to learn a shared discriminative representation across modalities. This module leverages optimal transport theory to precisely characterize the minimal transport cost between distributions. Compared with symmetric measures such as KL divergence, the Wasserstein distance provides stable gradient guidance even when there is a large distribution shift in the feature space, thus ensuring effective cross modal alignment. Second, to prevent excessive modal convergence caused by the above constraint, the semantic adversarial module treats each modality as an independent domain, assigns modality origin labels to samples, and trains a domain classifier. This classifier forms an adversarial game with the similarity module: the similarity module strives to confuse modal differences, while the domain classifier tries to distinguish the source of each modality. As a result, while keeping inter class distributions close, the model precisely preserves the unique deep information of each modality. This adversarial design is introduced for the first time in the field of trustworthy multimodal classification and effectively resolves the conflict between feature collapse and information loss.Finally, the confidence supervised self attention fusion module maps each modality’s features into queries, keys, and values, computes inter modal dependency weights via multi head self attention, and uses the True Class Probability (TCP) as a trustworthy supervision signal to train a confidence estimation branch, enabling the model to dynamically evaluate the evidence quality of each modality during testing without requiring labels. The fusion loss combines classification cross entropy and confidence mean square error, driving the attention weights to be adjusted in real time according to current confidence, thereby enhancing high quality modalities and suppressing low quality or noisy ones. This closed loop supervision mechanism allows uncertainty measures to directly participate in the generation of fusion weights, breaking the limitations of traditional static fusion. Experiments on five public multimodal datasets (Cub, Caltech101, Scene15, Animal, and PIE) demonstrate that JALDF achieves leading accuracy results across all datasets. Specifically, it attains 81.40% on Scene15, outperforming the suboptimal RMVC by 8.35 percentage points, and reaches 95.14% on PIE, significantly surpassing existing methods. Additional experiments on hyperparameter analysis, conflict sensitivity, uncertainty estimation, ablation, and visualization further validate the effectiveness and superiority of JALDF from multiple perspectives. In summary, through the organic synergy of adversarial like learning and dynamic fusion, JALDF effectively solves the three major problems of overconfidence, feature assimilation, and rigid fusion in multimodal classification, offering a highly accurate and robust technical solution for trustworthy multimodal classification.

摘要： 可信的多模态分类通过整合异构数据流并量化预测不确定性，在医疗诊断、自动驾驶等高风险领域具有重要意义。然而，现有方法在处理多模态冲突时，多数方法仍存在三个核心问题：机械噪声处理导致模型在证据匮乏时过度自信、统一融合范式造成模态特有深层语义被“特征同化”而丢失、以及不确定性估计未能作为动态反馈信号闭环引导融合决策。为应对这些核心挑战，本文提出联合类对抗学习与动态融合的可信多模态分类方法（JALDF）。该方法旨在通过特征对齐、语义保留和自适应融合的协同优化，系统性提升多模态分类的准确性、抗噪性和可信性。 在技术实现上，JALDF包含三个核心创新模块。首先，跨模态类相似度模块对各模态特征进行Sigmoid稀疏化和L1正则化以筛选高判别力子集，随后按类别提取样本，采用Wasserstein距离度量同一类别在不同模态下特征分布的差异，通过最小化该距离迫使网络建立跨模态共享的判别性表征。该模块利用最优传输理论精确刻画分布间的最小搬运代价，相较于KL散度等对称性度量，Wasserstein距离在特征空间存在较大分布偏移时仍能提供稳定的梯度指导，从而确保跨模态对齐的有效性。其次，为抑制上述约束导致的模态过度趋同，语义对抗类模块将每个模态视作独立领域，为样本分配模态归属标签并训练领域分类器，该分类器与相似度模块形成对抗博弈，即相似度模块努力混淆模态差异，而领域分类器则力图区分模态来源，从而在保持类间分布接近的同时，精准保留各模态独有的深层信息。这一对抗性设计首次被引入多模态可信分类领域，有效化解了特征塌缩与信息丢失的矛盾。最后，置信度监督的自注意力融合模块将各模态特征映射为查询、键和值，通过多头自注意力计算模态间依赖权重，并以真实类概率（TCP）作为可信监督信号训练置信度估计分支，使模型在测试阶段无需标签即可动态评估各模态证据质量；融合损失同时包含分类交叉熵与置信度均方误差，驱动注意力权重根据当前置信度实时调整，实现对高质量模态的增强和低质量、噪声模态的抑制。该闭环监督机制使得不确定性度量直接参与融合权重的生成，突破了传统静态融合的局限。 在五个公开多模态数据集（Cub、Caltech101、Scene15、Animal、PIE）上的实验表明，JALDF在准确率上均取得领先结果，其中Scene15上达81.40%，较次优模型RMVC提升8.35个百分点，PIE上达95.14%，显著优于现有方法。其余超参数分析、冲突数据敏感性、不确定性估计、消融及可视化等实验进一步地从多个维度验证了JALDF的有效性和先进性。 综上，JALDF通过类对抗学习与动态融合的有机协同，有效解决了多模态分类中的过度自信、特征同化和融合刚性三大难题，为可信多模态分类提供了兼具高精度与强鲁棒性的技术方案。