摘要： 多模态视觉-语言基础模型在医学领域展现出重要应用潜力，但由于医学数据语义结构复杂、跨模态关系建模困难，现有方法仍存在明显不足：一方面，基于患者的刚性对齐策略忽略语义相似性，导致不合理的负样本排斥，影响模型学习能力；另一方面，缺乏对报告与影像多层级语义结构的统一建模，难以实现细粒度的跨模态层次化对齐。针对上述问题，该文提出了一种语义驱动的全局–局部层级对齐的医学视觉语言分类模型（GLCA），通过全局–局部协同对齐实现更优的医学视觉语言分类模型。具体而言，GLCA包括语义驱动的患者间软全局对齐和渐进式三粒度患者内局部对齐两部分。语义驱动的患者间软全局对齐利用跨患者语义样本对挖掘和相关性加权对比惩罚来构建更连续、更符合真实语义关系的特征空间。渐进式三粒度患者内局部对齐通过渐进式查询融合策略在三个层次上对齐视觉与文本特征：粗粒度（报告–图像）、中粒度（句子–区域）、细粒度（词语–图块），实现跨模态与跨粒度的有效交互。其中，全局–局部协同对齐首先利用患者间的软全局对齐构建符合真实语义关系的特征空间，进而通过患者内的渐进式三粒度局部对齐实现视觉与文本特征的逐层匹配，两者协同优化，确保跨模态语义的连续嵌入与精准对应。在四个胸部X光数据集上进行的大量实验表明，GLCA在零样本预测分类和少样本微调分类任务中均显著优于现有方法。其中，对于公开的14分类胸部X光片数据集ChestXray14上的零样本预测分类实验，在AUC、F1和ACC指标上分别较次优方法提升了1.2%、2.0%和2.2%。

Abstract: Multimodal vision-language foundation models show great potential in the medical domain, yet face notable limitations due to complex medical semantics and challenging cross-modal modeling. Patient-level rigid alignment ignores semantic similarity, causing unreasonable negative repulsion and degrading learning, while the lack of unified hierarchical modeling between reports and images hinders fine-grained cross-modal alignment. To address the above issues, this paper proposes a global-local collaborative alignment (GLCA), which achieves an improved medical vision-language classification model. GLCA consists of two main components: semantic-driven cross-patient soft global alignment and progressive three-granularity intra-patient local alignment. The semantic-driven cross-patient soft global alignment leverages cross-patient semantic sample pair mining and correlation-weighted contrastive penalty to construct a more continuous feature space that better reflects authentic semantic relationships. The progressive three-granularity intra-patient local alignment aligns visual and textual features at three levels-coarse (report-image), mid (sentence-region), and fine (word-patch)-via progressive query fusion, enabling effective cross-modal interaction. Global-local collaborative alignment first builds a semantically consistent feature space through inter-patient soft global alignment, then performs layer-wise matching via intra-patient multi-granularity alignment, ensuring continuous and precise cross-modal semantic correspondence. Extensive experiments are conducted on four chest X-ray datasets. The results demonstrate that GLCA significantly outperforms existing methods in both zero-shot prediction classification and few-shot fine-tuning classification tasks. On the public 14-class ChestXray14 dataset, the zero-shot prediction classification achieves improvements of 1.2%, 2.0%, and 2.2% over the second-best method in terms of AUC, F1, and ACC, respectively.