Abstract: Video captioning aims to deeply analyze video content and accurately and fluently describe it in natural language. Concepts corresponding to the objects, actions, and attributes of video content can serve as media for video captioning. Although some studies have explored concept-guided video captioning, two major issues remain: limited concept-detection accuracy and insufficient concept utilization. To address these issues, this study proposes a Multimodal Video Captioning approach guided by Global and local Concepts (CGMVC) to improve the quality of generated descriptions. First, the multimodal features of the videos are extracted using different backbone networks. It leverages the HMMC model via Hierarchical Matching Video-to-Text Retrieval (HMVT2R) to obtain textual information from videos. Subsequently, it uses Multimodal Feature Fusion and a Concept-Detection (M2FCD) network to precisely detect concepts. To fully utilize the detected concepts, a Concept Projection Module (CPM) is employed to uncover the latent themes of videos to globally guide decoding and the Semantic Attention Module (SAM) and Cross-Attention Module (CAM) are used to locally optimize decoding by leveraging concepts and multimodal features of videos. By fully utilizing concepts and information from different modalities, more natural and accurate descriptions can be generated. Experiments on the MSVD and MSR-VTT datasets show that the CGMVC model achieves CIDEr scores of 111.2% and 64.1% and BLEU@4 scores of 57.1% and 51.2%, respectively. Comparative and ablation studies demonstrate the superiority of the CGMVC method over baseline approaches and other state-of-the-art methods.

Key words: video captioning, concept guidance, concept detection, video-to-text retrieval, Semantic Attention Module (SAM)

摘要： 视频描述旨在深入分析视频内容，用自然语言准确、流畅地描述视频内容。概念对应于视频内容中的对象、动作和属性，可以作为视频描述的媒介。虽然已经有部分研究使用概念引导视频描述，但是仍然存在两个主要问题：概念检测精度有限和概念利用率不足。针对这些问题，提出了全局和局部概念引导的多模态视频描述(CGMVC)方法，用于提高生成描述的质量。首先用不同的骨干网络提取视频的多模态特征，利用HMMC模型通过分层匹配的视频到文本检索(HMVT2R)提供视频的文本信息，然后使用多模态特征融合和概念检测(M2FCD)网络精确检测概念。为了充分利用检测到的概念，通过概念投影模块(CPM)挖掘视频的潜在主题从全局层面引导解码，通过语义注意力模块(SAM)和交叉注意力模块(CAM)分别利用概念和视频的多模态特征，实现局部层面的解码优化。通过充分利用概念和不同模态的信息，生成更加自然和准确的描述。在MSVD和MSR-VTT数据集上的实验结果表明，CGMVC模型的CIDEr和BLEU@4分别达到了111.2%、57.1%和 64.1%、51.2%，CGMVC方法相对于基线方法和其他先进方法具有一定的优越性。

关键词: 视频描述, 概念引导, 概念检测, 视频到文本检索, 语义注意力模块

CLC Number: 

• TP391