Abstract: Vision-Language Models (VLMs) have demonstrated exceptional capabilities in cross-modal understanding by aligning Large Language Models (LLMs) with visual encoders. However, cross-modal hallucination remains a critical bottleneck that limits their deployment in accuracy-sensitive scenarios. The fundamental cause lies in the fact that the underlying LLM base, pre-trained on massive text corpora, establishes strong language priors that often override authentic visual features during autoregressive decoding. A detailed analysis of the internal attention mechanism during decoding reveals significant hierarchical disparities in visual feature evolution: shallow layers exhibit highly diffuse attention containing substantial redundant noise, which easily triggers attention drift, while deep layers suffer from severe attention decay due to language dominance, effectively severing the link between text generation and visual facts. Existing decoding-time intervention methods typically adopt global, coarse-grained strategies, failing to systematically address the contradiction between shallow-layer diffusion and deep-layer decay. Consequently, this paper proposes a Multi-layer Cumulative Visual Prior (MCVP) method, which reconstructs feature distributions at the decoding stage through three synergistic mechanisms. First, a sparsified focusing strategy is introduced to dynamically evaluate shallow-layer responses and filter background noise to extract high-confidence features, fundamentally preventing attention drift. Second, a layer-wise recursive accumulation mechanism is designed to construct a temporally robust global visual prior by applying weighted aggregation to preceding shallow-layer sparse features via a sliding window. Finally, an adaptive soft injection strategy is proposed to calculate the real-time visual dependency of the current generation step, dynamically incorporating the robust prior into deep-layer cross-attention modules to rebalance modal weights and enforce strict alignment with visual facts. Extensive empirical evaluations on LLaVA-1.5, MiniGPT4, and DeepseekVL, based on CHAIR and POPE benchmarks, demonstrate that MCVP significantly outperforms baseline sampling and state-of-the-art intervention methods. In the CHAIR benchmark, MCVP achieves a relative improvement of 18.8% in hallucination suppression performance; specifically, it reduces the sentence-level hallucination rate of LLaVA-1.5 from 48.4% to 38.2% while maintaining a high recall of 77.7%, successfully overcoming the tendency of traditional exploratory strategies to over-fit language probabilities and achieving an optimal balance between precision and coverage. In POPE testing, MCVP achieves a state-of-the-art F1 score of 88.45% on MiniGPT4 and effectively corrects the over-prediction issues caused by shallow-layer diffusion in DeepseekVL. Comprehensive ablation and visualization analyses confirm that these mechanisms effectively prevent deep-layer feature decay and refocus attention onto target regions. In summary, MCVP accurately captures the evolutionary patterns of hierarchical attention and provides an efficient, training-free approach to suppressing cross-modal hallucinations without compromising general reasoning capabilities, establishing a new theoretical and practical paradigm for white-box hierarchical intervention in large model optimization.

摘要： 视觉语言模型（VLMs）通过对齐大语言模型与视觉编码器，在跨模态任务中展现出卓越能力，但跨模态幻觉问题严重限制了其在严谨场景中的应用。其根本原因在于，大语言模型基座在纯文本预训练中固化了强烈的语言先验，在自回归解码时往往反客为主压制真实的视觉特征。深入剖析解码阶段的内部注意力机制发现，视觉特征在不同网络层级呈现显著的演化差异：浅层网络视觉注意力高度弥散，包含大量易诱发注意力漂移的冗余噪声；而深层网络受语言先验主导，跨模态注意力严重衰退，阻断了文本生成与视觉事实的关联。现有解码干预方法多采用全局粗粒度策略，难以系统应对“浅层弥散、深层衰退”的矛盾。因此，本研究提出一种基于多层累积视觉先验增强（Multi-layer Cumulative Visual Prior, MCVP）的幻觉抑制方法，通过三大协同机制在解码阶段精细化重构特征分布。首先，引入稀疏化聚焦策略动态评估浅层响应，过滤背景噪声以提炼高置信度特征，从根源避免注意力漂移；其次，设计层级递归累积机制，利用滑动窗口对前序浅层稀疏化注意力进行加权聚合，构建时序稳健的全局视觉先验；最后，提出自适应软注入策略，实时计算当前生成阶段的视觉依赖度，将稳健先验动态融入深层交叉注意力模块，强制重塑模态权重以严格对齐视觉事实。在LLaVA-1.5、MiniGPT4和DeepseekVL等模型上，基于CHAIR与POPE基准的实证评估表明，MCVP相较于基础采样及现有先进干预方法优势显著。在CHAIR测试中，MCVP使综合幻觉抑制性能平均相对提升18.8%，其中LLaVA-1.5的句子级幻觉率从48.4%大幅降至38.2%，且在削减幻觉的同时维持了77.7%的高召回率，克服了传统探索性策略易拟合语言概率的缺陷，实现了更优的精度与覆盖率平衡。在POPE测试中，MCVP在MiniGPT4上取得88.45%的最优F1分数，并在DeepseekVL上有效修正了因浅层弥散导致的过度预测问题。深入的消融与可视化分析亦证实，所提机制成功防止了深层特征衰退，并将注意力重新紧密聚焦于目标区域。综上所述，MCVP精准捕捉了层级注意力的演化规律，在不损害通用推理能力且无需任何额外训练成本的前提下，高效、精准地抑制了跨模态幻觉，为未来基于白盒层级干预的免训练大模型优化奠定了新的理论与实践范式。