Weakly Supervised Video Anomaly Detection Method for Rope Skipping Combined with Semantic Prompts and Memory Enhancement

doi:10.19678/j.issn.1000-3428.0069415

Abstract

Abstract:

The student evaluation enhancement platform collects millions of jump rope movement data points from primary and secondary school students for physical fitness assessment. However, in the collected jump rope videos, there are various abnormal videos that do not meet the shooting requirements, such as non-jump rope videos, and characters that do not appear in full body. This seriously affects the accuracy and robustness of the follow-up physical fitness assessment model of students. To solve these problems, this study proposes a weakly supervised video anomaly detection method that combines semantic cues and memory enhancement. First, the visual features of normal and abnormal skipping rope videos are extracted, and the normal and abnormal features are trained in pairs to enhance the model's perception of abnormal video features. Second, two self-supervised memory networks are designed to store and separate the features of normal and abnormal videos to further enhance the feature representation ability of the model. Finally, a prompt learning method is introduced to transfer the semantic prior knowledge of various skip rope exception types in a large-scale pre-training model to enhance the model's understanding of the semantic information of various exception types in the case of insufficient samples. The experimental results show that the AUC of the proposed method on the self-built Skip Rope Anomaly Detection(SRAD) dataset is 94.14%, which is 2.71 percentage points higher than that of the benchmark method, thereby exhibiting high accuracy. The proposed method is of great significance for realizing intelligent physical fitness evaluation and promoting educational assessment reform.

Key words: video anomaly detection, prompt learning, memory network, weakly supervised learning, physical fitness assessment

摘要：

面向全国的学生综合评价与发展平台采集了百万级学生跳绳运动数据用于中小学生身体素质测评, 然而在采集的跳绳视频中存在着非跳绳视频、人物未全身出镜等不符合拍摄要求的各类异常视频, 严重影响了后续学生身体素质测评模型的准确性和鲁棒性。针对上述问题, 提出一种联合语义提示和记忆增强的弱监督跳绳视频异常检测方法。该方法首先提取正常跳绳视频和异常跳绳视频的视觉特征, 将正常特征和异常特征成对训练, 增强模型对异常视频特征的感知能力; 其次设计两个自监督记忆网络分别存储和分离正常视频和异常视频的特征, 进一步增强模型的特征表达能力; 最后引入提示学习方法迁移大规模预训练模型中的多种跳绳异常类型的语义先验知识, 增强模型在样本不足的情况下对多种异常类型语义信息的理解。实验结果表明, 该方法在自建的跳绳异常检测数据集(SRAD)上的AUC为94.14%, 相较于基准方法提升了2.71个百分点, 具有较高的准确性。该方法对实现身体素质的智能测评、推动教育评价改革具有重要意义。

关键词: 视频异常检测, 提示学习, 记忆网络, 弱监督学习, 身体素质测评

Xuanyu ZHOU, Lianhua WU, Qinhua ZHENG, Tianxing XIAO, Zixuan WANG, Simin ZHANG. Weakly Supervised Video Anomaly Detection Method for Rope Skipping Combined with Semantic Prompts and Memory Enhancement[J]. Computer Engineering, 2024, 50(7): 87-95.

周炫余, 吴莲华, 郑勤华, 肖天星, 王紫璇, 张思敏. 联合语义提示和记忆增强的弱监督跳绳视频异常检测方法[J]. 计算机工程, 2024, 50(7): 87-95.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0069415

https://www.ecice06.com/EN/Y2024/V50/I7/87

Figures/Tables 10

Fig.1 Overall framework of weakly supervised video anomaly detection method for rope skipping combining with semantic prompts and memory enhancement

Fig.2 Architecture of dual memory network

Fig.3 Example of prompt messages

Fig.4 Example of abnormal jump rope frame

Fig.5 Hyperparameter experiment results on SRAD dataset

References 26

1	中华人民共和国国务院. 深化新时代教育评价改革总体方案[EB/OL]. [2023-10-12]. http://www.moe.gov.cn/jyb_xxgk/moe_1777/moe_1778/202010/t20201013_494381.html.
	State council of the PRC. Overall plan for deepening the reformof educational evaluation in the new era[EB/OL]. [2023-10-12]. http://www.moe.gov.cn/jyb_xxgk/moe_1777/moe_1778/202010/t20201013_494381.html. (in Chinese)
2	教育部办公厅. 关于开展信息技术支撑学生综合素质评价试点工作的通知[EB/OL]. [2023-10-12]. http://www.moe.gov.cn/srcsite/A16/s3342/202211/t20221118_995825.html.
	General Office of the Ministry of Education. Notice on carrying out the pilot work of information Technology Supporting the evaluation of students' comprehensive quality[EB/OL]. [2023-10-12]. http://www.moe.gov.cn/srcsite/A16/s3342/202211/t20221118_995825.html. (in Chinese)
3	丁小欧, 于晟健, 王沐贤, 等. 基于相关性分析的工业时序数据异常检测. 软件学报, 2020, 31(3): 726- 747. URL
	DING X O, YU S J, WANG M X, et al. Anomaly detection on industrial time series based on correlation analysis. Journal of Software, 2020, 31(3): 726- 747. URL
4	严莉, 张凯, 徐浩, 等. 基于图注意力机制和Transformer的异常检测. 电子学报, 2022, 50(4): 900- 908. URL
	YAN L, ZHANG K, XU H, et al. Abnormal detection based on graph attention mechanisms and transformer. Acta Electronica Sinica, 2022, 50(4): 900- 908. URL
5	肖进胜, 郭浩文, 谢红刚, 等. 监控视频异常行为检测的概率记忆自编码网络. 软件学报, 2023, 34(9): 4362- 4377. URL
	XIAO J S, GUO H W, XIE H G, et al. Probabilistic memory auto-encoding network for abnormal behavior detection in surveillance videos. Journal of Software, 2023, 34(9): 4362- 4377. URL
6	朱红蕾, 朱昶胜, 徐志刚. 人体行为识别数据集研究进展. 自动化学报, 2018, 44(6): 978- 1004. URL
	ZHU H L, ZHU C S, XU Z G. Research advances on human activity recognition datasets. Acta Automatica Sinica, 2018, 44(6): 978- 1004. URL
7	杨静, 吴成茂, 周流平. 基于全局-局部自注意力网络的视频异常检测方法. 通信学报, 2023, 44(8): 241- 250. URL
	YANG J, WU C M, ZHOU L P. Novel video anomaly detection method based on global-local self-attention network. Journal on Communications, 2023, 44(8): 241- 250. URL
8	何平, 李刚, 李慧斌. 基于深度学习的视频异常检测方法综述. 计算机工程与科学, 2022, 44(9): 1620- 1629. doi: 10.3969/j.issn.1007-130X.2022.09.012
	HE P, LI G, LI H B. A survey on deep learning based video anomaly detection. Computer Engineering & Science, 2022, 44(9): 1620- 1629. doi: 10.3969/j.issn.1007-130X.2022.09.012
9	CHONG Y S, TAY Y H. Abnormal event detection in videos using spatiotemporal autoencoder[C]∥Proceedings of International Symposium on Neural Networks. Berlin, Germany: Springer, 2017: 189-196.
10	YE M C, PENG X J, GAN W H, et al. AnoPCN: video anomaly detection via deep predictive coding network[C]∥Proceedings of the 27th ACM International Conference on Multimedia. New York, USA: ACM Press, 2019: 1805-1813.
11	LÜ H, CHEN C, CUI Z, et al. Learning normal dynamics in videos with meta prototype network[C]∥Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 15425-15434.
12	ZAHEER M Z, MAHMOOD A, KHAN M H, et al. Generative cooperative learning for unsupervised video anomaly detection[C]∥Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 14744-14754.
13	SULTANI W, CHEN C, SHAH M. Real-world anomaly detection in surveillance videos[C]∥Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 6479-6488.
14	ZHU Y, NEWSAM S. Motion-aware feature for improved video anomaly detection[C]∥Proceeding of the 30th British Machine Vision Conference. Cardiff, UK: British Machine Vision Association, 2019(19): 1-12.
15	ZHOU H, YU J Q, YANG W. Dual memory units with uncertainty regulation for weakly supervised video anomaly detection. Artificial Intelligence, 2023, 37(3): 3769- 3777.
16	ZHANG J G, QING L Y, MIAO J. Temporal convolutional network with complementary inner bag loss for weakly supervised anomaly detection[C]∥Proceedings of IEEE International Conference on Image Processing. Washington D. C., USA: IEEE Press, 2019: 4030-4034.
17	ZAHEER M Z, MAHMOOD A, ASTRID M, et al. CLAWS: clustering assisted weakly supervised learning with normalcy suppression for anomalous event detection[C]∥Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 358-376.
18	WU P, LIU J. Learning causal temporal relation and feature discrimination for anomaly detection. IEEE Transactions on Image Processing, 2021, 30, 3513- 3527. doi: 10.1109/TIP.2021.3062192
19	LI S, LIU F, JIAO L C. Self-training multi-sequence learning with transformer for weakly supervised video anomaly detection. Artificial Intelligence, 2022, 36(2): 1395- 1403.
20	WANG M M, XING J Z, MEI J B, et al. ActionCLIP: adapting language-image pretrained models for video action recognition. IEEE Transactions on Neural Networks and Learning Systems, 2024, 27(1): 1- 13.
21	YAO L W, HAN J H, WEN Y P, et al. DetCLIP: dictionary-enriched visual-concept paralleled pre-training for open-world detection[EB/OL]. [2024-02-10]. http://arxiv.org/abs/2209.09407v2.
22	MILLER G A. WordNet. Communications of the ACM, 1995, 38(11): 39- 41. doi: 10.1145/219717.219748
23	PU Y J, WU X Y, YANG L L, et al. Learning prompt-enhanced context features for weakly-supervised video anomaly detection[EB/OL]. [2024-02-10]. http://arxiv.org/abs/2306.14451v2.
24	SPEER R, CHIN J, HAVASI C. ConceptNet 5.5: an open multilingual graph of general knowledge. Artificial Intelligence, 2017, 31(1): 33- 42.
25	WASIM S T, NASEER M, KHAN S, et al. Vita-CLIP: video and text adaptive CLIP via multimodal prompting[C]∥Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2023: 23034-23044.
26	RADFORD A, KIM J W, HALAACY C, et al. Learning transferable visual models from natural language supervision[C]∥Proceedings of the 38th International Conference on Machine Learning Research. New York, USA: ACM Press, 2021: 139: 8748-8763.

[1]	YANG Dongju, HUANG Juntao. Chinese Scientific Literature Annotation Method Based on Large Language Model [J]. Computer Engineering, 2024, 50(9): 113-120.
[2]	QU Xiaoya, LI Bing, WEN Liqiang. Research on Event Extraction for Administrative Law Enforcement Case Texts [J]. Computer Engineering, 2024, 50(9): 63-71.
[3]	Biqing ZENG, Pengfei CHEN, Yongtao YAO. Aspect Sentiment Triplet Extraction Combining Chain-of-Thought and Low-Rank Adaptation Fine-Tuning [J]. Computer Engineering, 2024, 50(7): 53-62.
[4]	DENG Yuanfei, LI Jiawei, JIANG Yuncheng. Similarity Computation of Patent Phrases Based on Knowledge Injection Prompt Learning [J]. Computer Engineering, 2024, 50(4): 294-302.
[5]	Hongpeng LI, Bo MA, Yating YANG, Lei WANG, Zhen WANG, Xiao LI. Document-level Event Extraction Method Based on Slot Semantic Enhanced Prompt Learning [J]. Computer Engineering, 2023, 49(9): 23-31.
[6]	ZHANG Boxu, PU Zhi, CHENG Xi. Research on Uyghur Text Classification Based on Prompt Learning [J]. Computer Engineering, 2023, 49(6): 292-299,313.
[7]	HUANG Tao, WU Kaijun, WANG Dicong, BAI Chenshuai, TAO Xiaomiao. Video Anomaly Detection Based on Improved Time Segmentation Network [J]. Computer Engineering, 2022, 48(11): 137-144.
[8]	GU Ping, QIU Jiatao, LUO Changjiang, ZHANG Zhipeng. Video Anomaly Detection Algorithm Based on Object Spatio-Temporal Context Fusion [J]. Computer Engineering, 2022, 48(10): 169-175.
[9]	LOU Yingxi, YUAN Wenhao, PENG Rongqun. Speech Enhancement Method Based on Quasi Recurrent Neural Network [J]. Computer Engineering, 2020, 46(4): 316-320.
[10]	JING Zhuangwei, GUAN Haiyan, PENG Daifeng, YU Yongtao. Survey of Research in Image Semantic Segmentation Based on Deep Neural Network [J]. Computer Engineering, 2020, 46(10): 1-17.
[11]	PENG Yuqing,SONG Chubai,YAN Qian,ZHAO Xiaosong,WEI Ming. Research on Chinese Text Classification Based on Hybrid Model of VDCNN and LSTM [J]. Computer Engineering, 2018, 44(11): 190-196.
[12]	MENG Kui,LIU Mengchi,HU Jie. Query Intention Recognition Model Based on Character Level Cyclic Network [J]. Computer Engineering, 2017, 43(3): 181-186.
[13]	ZHOU Hong-zhi, CHENG Xiang-yang. A Video Anomaly Detection Scheme Based on Local Spatio-temporal Features [J]. Computer Engineering, 2014, 40(4): 203-208,213.
[14]	TANG Liang-Rong, JIANG Zhen, XU Wei-Hong, LI Ying. Robustness Analysis of Parameterized Fuzzy Associative Memory Network [J]. Computer Engineering, 2010, 36(10): 212-214.
[15]	GAO Lin; LIU Ji-ming; ZHANG Shi-wu; YANG Jie. Dynamic Changes of Social Network Structure Caused by Memory Characteristics [J]. Computer Engineering, 2010, 36(10): 11-13.

Please choose a citation manager

Content to export