视频文本语义对齐与全视频依赖的弱监督时序动作定位

doi:10.19678/j.issn.1000-3428.0252238

摘要/Abstract

摘要： 针对现有弱监督时序动作定位(WTAL)研究存在的未充分利用动作的时序特性、全局特性和动作语义一致性等问题,提出视频文本语义对齐与全视频依赖的方法(FVD-ALM),充分利用多源信息以提升动作定位的准确性和鲁棒性。首先,依托膨胀卷积扩大模型的感受野,结合注意力机制对视频内动作的变化实施精确的特征强化,确保获得准确的时序特征,捕捉动作的动态变化。随后,采用基于高斯混合模型(GMM)的期望最大化(EM)算法提取并强化视频中的全局信息,生成精确的时序激活图,理解视频的整体内容,辅助动作的定位过程。最后,设计视频文本语义对齐模块,结合动作标签中的文本信息全面理解动作,训练模型补全描述动作的文本信息,增强模型对动作类别一致性的认知并有效区分不同动作类别。实验结果表明,在THUMOS14和ActivityNet1.3这两个主流数据集上,该方法均有效,其中在THUMOS14上实现了39.1%的均值平均精度(mAP),比DTRP-Loc方法提高了2.0百分点,证实了结合多源信息的方法能够显著提高动作定位的准确性,为WTAL任务提供了一种有效的解决方案。

关键词: 弱监督动作定位, 视频文本语义对齐, 全视频依赖, 伪标签生成, 多模态融合

Abstract: In response to the challenges in existing Weakly-supervised Temporal Action Localization (WTAL) research, such as the underutilization of action temporal characteristics, global properties, and action semantic consistency, a method called Weakly-Supervised Action Localization via Video—Text Semantic alignment and Full Video Dependency (FVD-ALM) is proposed. First, dilated convolution networks are used to expand the receptive field of the model, and attention mechanisms are utilized to precisely enhance the temporal features of action instances, ensuring accurate temporal feature extraction. Subsequently, an Expectation-Maximization (EM) algorithm based on a Gaussian Mixture Model (GMM) is applied to extract and enhance global information from the video, generating accurate temporal class activation maps to aid in the action localization process. Finally, a video—text semantic alignment module is designed to comprehensively understand actions by combining textual information with action labels. The model is trained to complete textual descriptions of actions, thereby enhancing its cognitive ability for action-category consistency and effectively distinguishing different action categories. Experimental results on the THUMOS14 and ActivityNet1.3 datasets confirm the effectiveness of this method, achieving an average mean Average Precision (mAP) of 39.1% on THUMOS14, which is a 2.0 percentage points improvement over the DTRP-Loc method. This demonstrates that the method of integrating multisource information significantly improves the accuracy of action localization and provides an effective solution for weakly-supervised action localization tasks.

Key words: Weakly-supervised Temporal Action Localization (WTAL), video—text semantic alignment, full video dependency, pseudo-label generation, multimodal fusion

中图分类号:

TP391.4

党伟超, 裴丽仙, 高改梅, 刘春霞. 视频文本语义对齐与全视频依赖的弱监督时序动作定位[J]. 计算机工程, 2026, 52(7): 354-367.

DANG Weichao, PEI Lixian, GAO Gaimei, LIU Chunxia. Weakly-Supervised Temporal Action Localization via Video—Text Semantic Alignment and Full Video Dependency[J]. Computer Engineering, 2026, 52(7): 354-367.

https://www.ecice06.com/CN/Y2026/V52/I7/354

参考文献

[1] CHENG F, BERTASIUS G. Tallformer: temporal action localization with long-memory transformer[C]//Proceedings of the 17th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2022: 503-521.
[2] TIRUPATTUR P, DUARTE K, RAWAT Y S, et al. Modeling multi-label action dependencies for temporal action localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA: IEEE Press, 2021: 1460-1470.
[3] ZHANG M, HU H Y, LI Z J. Temporal action localization with coarse-to-fine network[J]. IEEE Access, 2022, 10: 96378-96387.
[4] HE B, YANG X T, KANG L, et al. ASM-Loc: action-aware segment modeling for weakly-supervised temporal action localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE Press, 2022: 13915-13925.
[5] HU Y F, FU J, CHEN M Y, et al. Learning proposal-aware re-ranking for weakly-supervised temporal action localization[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(1): 207-220.
[6] 郭文斌, 杨兴明, 蒋哲远, 等. 多时间尺度一致性的弱监督时序动作定位[J]. 计算机工程与应用, 2023, 59(10): 151-161. GUO W B, YANG X M, JIANG Z Y, et al. Multi-temporal scales consensus for weakly supervised temporal action localization[J]. Computer Engineering and Applications, 2023, 59(10): 151-161. (in Chinese)
[7] 侯永宏, 李岳阳, 郭子慧. 基于对比学习的弱监督时序动作定位[J]. 天津大学学报, 2023, 56(1): 73-80. HOU Y H, LI Y Y, GUO Z H. Weakly supervised temporal action localization based on contrastive learning[J]. Journal of Tianjin University, 2023, 56(1): 73-80. (in Chinese)
[8] ZHOU J X, WU Y. Temporal feature enhancement dilated convolution network for weakly-supervised temporal action localization[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Waikoloa, USA: IEEE Press, 2023: 6017-6026.
[9] HUANG L J, WANG L, LI H S. Weakly supervised temporal action localization via representative snippet knowledge propagation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE Press, 2022: 3262-3271.
[10] QING Z W, SU H S, GAN W H, et al. Temporal context aggregation network for temporal action proposal refinement[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA: IEEE Press, 2021: 485-494.
[11] SRIDHAR D, QUADER N, MURALIDHARAN S, et al. Class semantics-based attention for action detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, Canada: IEEE Press, 2022: 13719-13728.
[12] ZHENG Z, WANG P, LIU W, LI J, et al. Distance-IoU loss: faster and better learning for bounding box regression[C]//Proceedings of the AAAI Conference on Artificial Intelligence.[S. l.]: AAAI Press, 2020: 12993-13000.
[13] HUANG J, KONG M, CHEN L Y, et al. Temporal RPN learning for weakly-supervised temporal action localization[C]// Proceedings of the 15th Asian Conference on Machine Learning. New York, USA: PMLR, 2024: 470-485.
[14] LIU Y Y, ZHU H, REN H H, et al. Fusion detection network with discriminative enhancement for weakly-supervised temporal action localization[J]. Expert Systems with Applications, 2024, 238: 122000.
[15] PAUL S, ROY S, ROY-CHOWDHURY A K. W-TALC: weakly-supervised temporal activity localization and classification[C]//Proceedings of the 15th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2018: 588-607.
[16] ZHANG C, CAO M, YANG D M, et al. CoLA: weakly-supervised temporal action localization with snippet contrastive learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA: IEEE Press, 2021: 16005-16014.
[17] GAO J Y, CHEN M Y, XU C S. Fine-grained temporal contrastive learning for weakly-supervised temporal action localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE Press, 2022: 19967-19977.
[18] LIU P, WANG C, QIN J, et al. Feature enhancement and foreground-background separation for weakly supervised temporal action localization[C]//Proceedings of the 5th ACM International Conference on Multimedia. New York, USA: ACM Press, 2024: 1-7
[19] QU S, CHEN G, LI Z, ZHANG L, LU F, KNOLL A. ACM-Net: action context modeling network for weakly-supervised temporal action localization[EB/OL].[2021-04-07]. https://arxiv.org/abs/2104.02967.
[20] HONG F T, FENG J C, XU D, et al. Cross-modal consensus network for weakly supervised temporal action localization[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York, USA: ACM Press, 2021: 1591-1599.
[21] JIANG H, TANG H Y, YAN M, et al. Revisiting unsupervised temporal action localization: the primacy of high-quality actionness and pseudolabels[C]//Proceedings of the 32nd ACM International Conference on Multimedia. New York, USA: ACM Press, 2024: 5643-5652.
[22] WANG C X, WANG J, XU W T. Double branch synergies with modal reinforcement for weakly supervised temporal action detection[J]. Journal of Visual Communication and Image Representation, 2024, 99: 104090.
[23] LI Z L, WANG Z L, LIU Q Y. Weakly supervised temporal action localization with actionness-guided false positive suppression[J]. Neural Networks, 2024, 175: 106307.
[24] YUN W, QI M, WANG C, MA H. Weakly-supervised temporal action localization by inferring salient snippet-feature[C]//Proceedings of the AAAI Conference on Artificial Intelligence.[S. l.]: AAAI Press, 2024: 6908-6916.
[25] YANG W F, ZHANG T Z, YU X Y, et al. Uncertainty guided collaborative training for weakly supervised temporal action detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA: IEEE Press, 2021: 53-63.
[26] LUO Z K, GUILLORY D, SHI B F, et al. Weakly-supervised action localization with expectation-maximization multi-instance learning[C]//Proceedings of the 16th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2020: 729-745.
[27] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the International Conference on Machine Learning (ICML). New York, USA: PMLR, 2021: 8748-8763.
[28] JU C, HAN T D, ZHENG K H, et al. Prompting visual-language models for efficient video understanding[C]//Proceedings of the 17th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2022: 105-124.
[29] LEI J, YU L C, BERG T L, et al. TVQA+: spatio-temporal grounding for video question answering[EB/OL].[2019-04-225]. https://arxiv.org/abs/1904.11574.
[30] CARREIRA J, ZISSERMAN A. Quo vadis, action recognition? A new model and the kinetics dataset[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE Press, 2017: 4724-4733.
[31] FEICHTENHOFER C. X3D: expanding architectures for efficient video recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, USA: IEEE Press, 2020: 200-210.
[32] ÇIÇEK Ö, ABDULKADIR A, LIENKAMP S S, et al. 3D U-Net: learning dense volumetric segmentation from sparse annotation[C]//Proceedings of the 19th Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Berlin, Germany: Springer, 2016: 424-432.
[33] LI X, ZHONG Z S, WU J L, et al. Expectation-maximization attention networks for semantic segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea: IEEE Press, 2020: 9166-9175.
[34] ALDOUS D J. Lower bounds for covering times for reversible Markov chains and random walks on graphs[J]. Journal of Theoretical Probability, 1989, 2(1): 91-100.
[35] PENNINGTON J, SOCHER R, MANNING C. GloVe: global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics, 2014: 1532-1543.
[36] LIN Z J, ZHAO Z, ZHANG Z, et al. Weakly-supervised video moment retrieval via semantic completion network[C]//Proceedings of the AAAI Conference on Artificial Intelligence.[S.l.]: AAAI Press, 2020: 11539-11546.
[37] IDREES H, ZAMIR A R, JIANG Y G, et al. The THUMOS challenge on action recognition for videos "in the wild"[J]. Computer Vision and Image Understanding, 2017, 155: 1-23.
[38] HEILBRON F C, ESCORCIA V, GHANEM B, et al. ActivityNet: a large-scale video benchmark for human activity understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA: IEEE Press, 2015: 961-970.
[39] LIN T W, ZHAO X, SU H S, et al. BSN: boundary sensitive network for temporal action proposal generation[C]//Proceedings of the 15th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2018: 3-21.
[40] LONG F C, YAO T, QIU Z F, et al. Gaussian temporal awareness networks for action localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE Press, 2020: 344-353.
[41] ZHAO P S, XIE L X, JU C, et al. Bottom-up temporal action localization with mutual regularization[C]//Proceedings of the 16th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2020: 539-555.
[42] CHEN M Y, GAO J Y, YANG S C, et al. Dual-evidential learning for weakly-supervised temporal action localization[C]//Proceedings of the 17th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2022: 192-208.
[43] TANG X J, FAN J S, LUO C C, et al. DDG-Net: discriminability-driven graph network for weakly-supervised temporal action localization[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Paris, France: IEEE Press, 2024: 6599-6609.
[44] REN H, YANG W F, ZHANG T Z, et al. Proposal-based multiple instance learning for weakly-supervised temporal action localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver, Canada: IEEE Press, 2023: 2394-2404.
[45] ZHANG S C, ZHAO C H. Cross-video contextual knowledge exploration and exploitation for ambiguity reduction in weakly supervised temporal action localization[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(6): 4568-4580.
[46] HUANG L J, WANG L, LI H S. Foreground-action consistency network for weakly supervised temporal action localization[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, Canada: IEEE Press, 2022: 7982-7991.
[47] LI J J, YANG T Y, JI W, et al. Exploring denoised cross-video contrast for weakly-supervised temporal action localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE Press, 2022: 19882-19892.
[48] 曹雨欣. 弱监督时序动作增量定位方法研究[D]. 西安: 西安理工大学, 2024. CAO Y X. Research on weakly supervised incremental local ization method for temporal actions[D]. Xi’an: Xi’an University of Technology, 2024. (in Chinese)
[49] ZHAO Y B, ZHANG H, GAO Z, et al. A snippets relation and hard-snippets mask network for weakly-supervised temporal action localization[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(8): 7202-7215.
[50] CHO K, VAN MERRIËNBOER B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics, 2014: 1724-1734.
[51] HOCHREITER S, SCHMIDHUBER J. Long short-term memory[J]. Neural Computation, 1997, 9(8): 1735-1780.

选择文件类型/文献管理软件名称

选择包含的内容