Abstract: Knowledge-based visual question answering requires world knowledge beyond images to generate answers. Existing studies mostly leverage multimodal large language models (MLLMs) to acquire visual reasoning knowledge, or employ large language models (LLMs) as knowledge engines to convert images into texts for reasoning. MLLMs boast prominent visual understanding capabilities but suffer from insufficient reasoning abilities, while LLMs excel at reasoning yet cannot directly parse visual information. How to give full play to the complementary advantages of the two types of models has not been fully explored in existing integration methods. To address the above issues, this paper proposes a novel knowledge-based visual question answering paradigm in which an LLM coordinates multiple MLLMs for collaborative reasoning. Multiple MLLMs serve as heterogeneous knowledge engines, and the LLM assumes the role of weighing and arbitrating multi-source evidence, realizing the deep integration of visual perception and logical reasoning capabilities. The framework consists of three core stages. First, in the question-aware caption generation stage, the BLIP-2 image-grounded text encoder combined with the GradCAM gradient localization method is adopted to accurately identify key visual regions in the image that are highly relevant to the question. Meanwhile, six types of complementary prompt templates are designed in light of the instruction response characteristics of two MLLMs, InstructBLIP and LLaVA-1.5, to generate multi-view, question-oriented perception captions, which provide rich visual context knowledge for subsequent reasoning. Second, in the candidate answer pool construction stage, the original image and the question are fed into two MLLMs respectively to generate independent candidate answers. This broadens the answer search space and effectively alleviates the knowledge blind spots and domain bias of a single model.Third, in the LLM-coordinated reasoning stage, perception captions, candidate answers and model identifiers are organized into structured prompts. Combined with the in-context learning mechanism to retrieve similar samples from the training set, the structured prompts are input into the LLaMA2-7B model. Relying on its built-in world knowledge and semantic reasoning ability, the LLM completes semantic correlation, conflict discrimination and credibility ranking of multi-source evidence, and finally outputs the optimal reasoning answer.Experimental results show that the proposed method achieves an accuracy of 69.4% on the OK-VQA dataset and 67.1% on the A-OKVQA direct answer task, delivering state-of-the-art performance among methods with the same parameter scale. Ablation experiments verify that removing the perception caption module and the candidate answer module leads to performance drops of 7.5 and 6.7 percentage points respectively. This fully demonstrates the necessity of multi-source visual evidence for LLM arbitration decision-making and the effectiveness of the multi-MLLM heterogeneous collaboration framework, providing new insights for building lightweight and efficient multimodal reasoning systems.

摘要： 知识型视觉问答需要借助图像之外的世界知识完成作答。现有研究多利用多模态大模型 (MLLM)获取视觉推理知识，或是以大语言模型 (LLM)作为知识引擎，将图像转化为文本以开展推理。MLLM视觉理解能力突出但推理能力不足，LLM推理能力强却无法直接解析视觉信息，如何充分发挥两类模型的互补优势，现有集成方法尚未得到充分研究。针对上述问题，本文提出一种 LLM 协调多个 MLLM 协同推理的知识型视觉问答新范式，将多个MLLM作为异构知识引擎，由 LLM 承担多源证据的权衡与仲裁角色，实现视觉感知能力与逻辑推理能力的深度融合。该框架包含三个核心阶段：其一，问题感知字幕生成阶段，采用BLIP-2图像关联文本编码器结合GradCAM梯度定位方法，精准识别图像中与问题高度相关的关键视觉区域；同时针对InstructBLIP与LLaVA-1.5两类MLLM的指令响应特性，设计6类互补性提示模板，生成多视角、问题导向的感知字幕，为后续推理提供丰富的视觉上下文知识。其二，候选答案池构建阶段，将原始图像与问题分别输入两个MLLM生成独立候选答案，拓宽答案搜索空间，有效缓解单模型的知识盲区与领域偏差问题。其三，LLM 协调推理阶段，将感知字幕、候选答案、模型标识整理为结构化提示，结合上下文学习机制检索训练集中的相似样本，输入LLaMA2-7B模型；LLM依托自身内置的世界知识与语义推理能力，完成多源证据的语义关联、冲突判别与可信度排序，最终输出最优推理答案。实验结果表明，所提方法在OK-VQA数据集上准确率可达69.4%，在A-OKVQA直接回答任务上准确率达67.1%，在同参数量级方法中取得最优性能。消融实验验证，移除感知字幕模块与候选答案模块会分别造成7.5、6.7个百分点的性能下降，充分证明了多源视觉证据对LLM仲裁决策的必要性，以及多MLLM异构协同框架的有效性，为构建轻量高效的多模态推理系统提供了新思路