视频文本语义对齐与全视频依赖的弱监督时序动作定位

doi:10.19678/j.issn.1000-3428.0252238

摘要/Abstract

摘要：

针对现有弱监督时序动作定位(WTAL)研究存在的未充分利用动作的时序特性、全局特性和动作语义一致性等问题, 提出视频文本语义对齐与全视频依赖的方法(FVD-ALM), 充分利用多源信息以提升动作定位的准确性和鲁棒性。首先, 依托膨胀卷积扩大模型的感受野, 结合注意力机制对视频内动作的变化实施精确的特征强化, 确保获得准确的时序特征, 捕捉动作的动态变化。随后, 采用基于高斯混合模型(GMM)的期望最大化(EM)算法提取并强化视频中的全局信息, 生成精确的时序激活图, 理解视频的整体内容, 辅助动作的定位过程。最后, 设计视频文本语义对齐模块, 结合动作标签中的文本信息全面理解动作, 训练模型补全描述动作的文本信息, 增强模型对动作类别一致性的认知并有效区分不同动作类别。实验结果表明, 在THUMOS14和ActivityNet1.3这两个主流数据集上, 该方法均有效, 其中在THUMOS14上实现了39.1%的均值平均精度(mAP), 比DTRP-Loc方法提高了2.0百分点, 证实了结合多源信息的方法能够显著提高动作定位的准确性, 为WTAL任务提供了一种有效的解决方案。

关键词: 弱监督动作定位, 视频文本语义对齐, 全视频依赖, 伪标签生成, 多模态融合

Abstract:

In response to the challenges in existing Weakly-supervised Temporal Action Localization (WTAL) research, such as the underutilization of action temporal characteristics, global properties, and action semantic consistency, a method called Weakly-Supervised Action Localization via Video—Text Semantic alignment and Full Video Dependency (FVD-ALM) is proposed. First, dilated convolution networks are used to expand the receptive field of the model, and attention mechanisms are utilized to precisely enhance the temporal features of action instances, ensuring accurate temporal feature extraction. Subsequently, an Expectation-Maximization (EM) algorithm based on a Gaussian Mixture Model (GMM) is applied to extract and enhance global information from the video, generating accurate temporal class activation maps to aid in the action localization process. Finally, a video—text semantic alignment module is designed to comprehensively understand actions by combining textual information with action labels. The model is trained to complete textual descriptions of actions, thereby enhancing its cognitive ability for action-category consistency and effectively distinguishing different action categories. Experimental results on the THUMOS14 and ActivityNet1.3 datasets confirm the effectiveness of this method, achieving an average mean Average Precision (mAP) of 39.1% on THUMOS14, which is a 2.0 percentage points improvement over the DTRP-Loc method. This demonstrates that the method of integrating multisource information significantly improves the accuracy of action localization and provides an effective solution for weakly-supervised action localization tasks.

Key words: Weakly-supervised Temporal Action Localization (WTAL), video—text semantic alignment, full video dependency, pseudo-label generation, multimodal fusion

党伟超, 裴丽仙, 高改梅, 刘春霞. 视频文本语义对齐与全视频依赖的弱监督时序动作定位[J]. 计算机工程, 2026, 52(7): 354-367.

DANG Weichao, PEI Lixian, GAO Gaimei, LIU Chunxia. Weakly-Supervised Temporal Action Localization via Video—Text Semantic Alignment and Full Video Dependency[J]. Computer Engineering, 2026, 52(7): 354-367.

https://www.ecice06.com/CN/Y2026/V52/I7/354

图/表 12

图1 整体框架

Fig.1 Overall framework

图2 Transformer重构器

Fig.2 Transformer reconstructor

图3 THUMOS14数据集上的定性结果可视化

Fig.3 Visualization of qualitative results on the THUMOS14 dataset

参考文献 51

1	CHENG F, BERTASIUS G. Tallformer: temporal action localization with long-memory transformer[C]//Proceedings of the 17th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2022: 503-521.
2	TIRUPATTUR P, DUARTE K, RAWAT Y S, et al. Modeling multi-label action dependencies for temporal action localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA: IEEE Press, 2021: 1460-1470.
3	ZHANG M , HU H Y , LI Z J . Temporal action localization with coarse-to-fine network. IEEE Access, 2022, 10, 96378- 96387. doi: 10.1109/ACCESS.2022.3205594
4	HE B, YANG X T, KANG L, et al. ASM-Loc: action-aware segment modeling for weakly-supervised temporal action localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE Press, 2022: 13915-13925.
5	HU Y F , FU J , CHEN M Y , et al. Learning proposal-aware re-ranking for weakly-supervised temporal action localization. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34 (1): 207- 220. doi: 10.1109/TCSVT.2023.3283430
6	郭文斌, 杨兴明, 蒋哲远, 等. 多时间尺度一致性的弱监督时序动作定位. 计算机工程与应用, 2023, 59 (10): 151- 161.
	GUO W B , YANG X M , JIANG Z Y , et al. Multi-temporal scales consensus for weakly supervised temporal action localization. Computer Engineering and Applications, 2023, 59 (10): 151- 161.
7	侯永宏, 李岳阳, 郭子慧. 基于对比学习的弱监督时序动作定位. 天津大学学报, 2023, 56 (1): 73- 80.
	HOU Y H , LI Y Y , GUO Z H . Weakly supervised temporal action localization based on contrastive learning. Journal of Tianjin University, 2023, 56 (1): 73- 80.
8	ZHOU J X, WU Y. Temporal feature enhancement dilated convolution network for weakly-supervised temporal action localization[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Waikoloa, USA: IEEE Press, 2023: 6017-6026.
9	HUANG L J, WANG L, LI H S. Weakly supervised temporal action localization via representative snippet knowledge propagation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE Press, 2022: 3262-3271.
10	QING Z W, SU H S, GAN W H, et al. Temporal context aggregation network for temporal action proposal refinement[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA: IEEE Press, 2021: 485-494.
11	SRIDHAR D, QUADER N, MURALIDHARAN S, et al. Class semantics-based attention for action detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, Canada: IEEE Press, 2022: 13719-13728.
12	ZHENG Z, WANG P, LIU W, LI J, et al. Distance-IoU loss: faster and better learning for bounding box regression[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S. l. ]: AAAI Press, 2020: 12993-13000.
13	HUANG J, KONG M, CHEN L Y, et al. Temporal RPN learning for weakly-supervised temporal action localization[C]// Proceedings of the 15th Asian Conference on Machine Learning. New York, USA: PMLR, 2024: 470-485.
14	LIU Y Y , ZHU H , REN H H , et al. Fusion detection network with discriminative enhancement for weakly-supervised temporal action localization. Expert Systems with Applications, 2024, 238, 122000. doi: 10.1016/j.eswa.2023.122000
15	PAUL S, ROY S, ROY-CHOWDHURY A K. W-TALC: weakly-supervised temporal activity localization and classification[C]//Proceedings of the 15th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2018: 588-607.
16	ZHANG C, CAO M, YANG D M, et al. CoLA: weakly-supervised temporal action localization with snippet contrastive learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA: IEEE Press, 2021: 16005-16014.
17	GAO J Y, CHEN M Y, XU C S. Fine-grained temporal contrastive learning for weakly-supervised temporal action localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE Press, 2022: 19967-19977.
18	LIU P, WANG C, QIN J, et al. Feature enhancement and foreground-background separation for weakly supervised temporal action localization[C]//Proceedings of the 5th ACM International Conference on Multimedia. New York, USA: ACM Press, 2024: 1-7
19	QU S, CHEN G, LI Z, ZHANG L, LU F, KNOLL A. ACM-Net: action context modeling network for weakly-supervised temporal action localization[EB/OL]. [2021-04-07]. https://arxiv.org/abs/2104.02967.
20	HONG F T, FENG J C, XU D, et al. Cross-modal consensus network for weakly supervised temporal action localization[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York, USA: ACM Press, 2021: 1591-1599.
21	JIANG H, TANG H Y, YAN M, et al. Revisiting unsupervised temporal action localization: the primacy of high-quality actionness and pseudolabels[C]//Proceedings of the 32nd ACM International Conference on Multimedia. New York, USA: ACM Press, 2024: 5643-5652.
22	WANG C X , WANG J , XU W T . Double branch synergies with modal reinforcement for weakly supervised temporal action detection. Journal of Visual Communication and Image Representation, 2024, 99, 104090. doi: 10.1016/j.jvcir.2024.104090
23	LI Z L , WANG Z L , LIU Q Y . Weakly supervised temporal action localization with actionness-guided false positive suppression. Neural Networks, 2024, 175, 106307. doi: 10.1016/j.neunet.2024.106307
24	YUN W, QI M, WANG C, MA H. Weakly-supervised temporal action localization by inferring salient snippet-feature[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S. l. ]: AAAI Press, 2024: 6908-6916.
25	YANG W F, ZHANG T Z, YU X Y, et al. Uncertainty guided collaborative training for weakly supervised temporal action detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA: IEEE Press, 2021: 53-63.
26	LUO Z K, GUILLORY D, SHI B F, et al. Weakly-supervised action localization with expectation-maximization multi-instance learning[C]//Proceedings of the 16th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2020: 729-745.
27	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the International Conference on Machine Learning (ICML). New York, USA: PMLR, 2021: 8748-8763.
28	JU C, HAN T D, ZHENG K H, et al. Prompting visual-language models for efficient video understanding[C]//Proceedings of the 17th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2022: 105-124.
29	LEI J, YU L C, BERG T L, et al. TVQA+: spatio-temporal grounding for video question answering[EB/OL]. [2019-04-225]. https://arxiv.org/abs/1904.11574.
30	CARREIRA J, ZISSERMAN A. Quo vadis, action recognition? A new model and the kinetics dataset[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE Press, 2017: 4724-4733.
31	FEICHTENHOFER C. X3D: expanding architectures for efficient video recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, USA: IEEE Press, 2020: 200-210.
32	ÇIÇEK Ö, ABDULKADIR A, LIENKAMP S S, et al. 3D U-Net: learning dense volumetric segmentation from sparse annotation[C]//Proceedings of the 19th Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI). Berlin, Germany: Springer, 2016: 424-432.
33	LI X, ZHONG Z S, WU J L, et al. Expectation-maximization attention networks for semantic segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea: IEEE Press, 2020: 9166-9175.
34	ALDOUS D J . Lower bounds for covering times for reversible Markov chains and random walks on graphs. Journal of Theoretical Probability, 1989, 2 (1): 91- 100. doi: 10.1007/BF01048272
35	PENNINGTON J, SOCHER R, MANNING C. GloVe: global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics, 2014: 1532-1543.
36	LIN Z J, ZHAO Z, ZHANG Z, et al. Weakly-supervised video moment retrieval via semantic completion network[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S. l. ]: AAAI Press, 2020: 11539-11546.
37	IDREES H , ZAMIR A R , JIANG Y G , et al. The THUMOS challenge on action recognition for videos "in the wild". Computer Vision and Image Understanding, 2017, 155, 1- 23.
38	HEILBRON F C, ESCORCIA V, GHANEM B, et al. ActivityNet: a large-scale video benchmark for human activity understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, USA: IEEE Press, 2015: 961-970.
39	LIN T W, ZHAO X, SU H S, et al. BSN: boundary sensitive network for temporal action proposal generation[C]//Proceedings of the 15th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2018: 3-21.
40	LONG F C, YAO T, QIU Z F, et al. Gaussian temporal awareness networks for action localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE Press, 2020: 344-353.
41	ZHAO P S, XIE L X, JU C, et al. Bottom-up temporal action localization with mutual regularization[C]//Proceedings of the 16th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2020: 539-555.
42	CHEN M Y, GAO J Y, YANG S C, et al. Dual-evidential learning for weakly-supervised temporal action localization[C]//Proceedings of the 17th European Conference on Computer Vision (ECCV). Berlin, Germany: Springer, 2022: 192-208.
43	TANG X J, FAN J S, LUO C C, et al. DDG-Net: discriminability-driven graph network for weakly-supervised temporal action localization[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Paris, France: IEEE Press, 2024: 6599-6609.
44	REN H, YANG W F, ZHANG T Z, et al. Proposal-based multiple instance learning for weakly-supervised temporal action localization[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver, Canada: IEEE Press, 2023: 2394-2404.
45	ZHANG S C , ZHAO C H . Cross-video contextual knowledge exploration and exploitation for ambiguity reduction in weakly supervised temporal action localization. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34 (6): 4568- 4580. doi: 10.1109/TCSVT.2023.3341881
46	HUANG L J, WANG L, LI H S. Foreground-action consistency network for weakly supervised temporal action localization[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, Canada: IEEE Press, 2022: 7982-7991.
47	LI J J, YANG T Y, JI W, et al. Exploring denoised cross-video contrast for weakly-supervised temporal action localization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE Press, 2022: 19882-19892.
48	曹雨欣. 弱监督时序动作增量定位方法研究[D]. 西安: 西安理工大学, 2024.
	CAO Y X. Research on weakly supervised incremental local ization method for temporal actions[D]. Xi'an: Xi'an University of Technology, 2024. (in Chinese)
49	ZHAO Y B , ZHANG H , GAO Z , et al. A snippets relation and hard-snippets mask network for weakly-supervised temporal action localization. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34 (8): 7202- 7215. doi: 10.1109/TCSVT.2024.3374870
50	CHO K, VAN MERRIËNBOER B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Doha, Qatar: Association for Computational Linguistics, 2014: 1724-1734.
51	HOCHREITER S , SCHMIDHUBER J . Long short-term memory. Neural Computation, 1997, 9 (8): 1735- 1780. doi: 10.1162/neco.1997.9.8.1735

[1]	崔爽锌, 卢搏, 张明月, 赵一汎, 王子铭, 刘新宇, 陈程立诏. 基于多模态融合的360°图像质量与美学评估方法[J]. 计算机工程, 2026, 52(6): 288-295.
[2]	王永旗, 王雷. 基于跨模态增强与时间步门控的多模态情感识别[J]. 计算机工程, 2026, 52(6): 258-267.
[3]	苏建华, 池云仙, 许云峰, 高凯. 基于注意力模态融合的多模态意图识别[J]. 计算机工程, 2026, 52(3): 234-242.
[4]	龙丽叶, 焦世超, 郭磊, 韩燮, 况立群. 基于紧凑中心的多模态三维模型检索研究[J]. 计算机工程, 2025, 51(2): 322-334.
[5]	冯赛赛, 葛东峰, 李涛, 刘一靖, 冀治航, 王琳, 张明川. 基于多模态融合的宫颈上皮内瘤变辅助诊断[J]. 计算机工程, 2025, 51(12): 304-310.
[6]	刘洋宏, 付杨悠然, 董性平. HDMapFusion: 用于自动驾驶的多模态融合高清地图生成(特邀)[J]. 计算机工程, 2025, 51(10): 18-26.
[7]	武星, 殷浩宇, 姚骏峰, 李卫民, 钱权. 面向视频数据的多模态情感分析[J]. 计算机工程, 2024, 50(6): 218-227.
[8]	朱贵德, 黄海. 文本视觉问答综述[J]. 计算机工程, 2024, 50(2): 1-14.
[9]	王文静, 范涛, 王国中, 赵海武. 基于SA-BPNN多模态融合的教学质量评价方法[J]. 计算机工程, 2024, 50(11): 390-398.
[10]	周昊玮, 刘勇, 玄萍. 基于预训练和多模态融合的假新闻检测[J]. 计算机工程, 2024, 50(1): 289-295.
[11]	吴志强, 解庆, 李琳, 刘永坚. 基于多模态融合的图神经网络推荐算法[J]. 计算机工程, 2024, 50(1): 91-100.
[12]	詹健浩, 甘利鹏, 毕永辉, 曾鹏, 李晓潮. 基于知识蒸馏的多模态融合行为识别方法[J]. 计算机工程, 2023, 49(10): 280-288, 297.
[13]	张相芬, 刘艳, 袁非牛. 基于倒金字塔深度学习网络的三维医学图像分割[J]. 计算机工程, 2022, 48(12): 304-311.
[14]	赵宏, 郭岚, 陈志文, 郑厚泽. 基于多模态融合与多层注意力的视频内容文本表述研究[J]. 计算机工程, 2022, 48(10): 45-54.
[15]	范裕莹, 李成娟, 易强, 李宝清. 基于改进TCN模型的野外运动目标分类[J]. 计算机工程, 2021, 47(9): 106-112.

选择文件类型/文献管理软件名称

选择包含的内容