面向视频行为识别深度模型的数据预处理方法

doi:10.19678/j.issn.1000-3428.0066795

摘要/Abstract

摘要：

以视频帧采样和数据增强为代表的预处理操作是提升视频行为识别深度模型性能的重要手段。针对现有视频数据预处理存在的采样视频帧区分性不足、数据增强方式单一等问题，提出一种面向视频行为识别深度模型的数据预处理方法。在视频帧采样上设计动作指导的片段化视频采样策略，综合考虑视频帧间差异特征与视频片段短期时序特征，通过显著行为动作获取关键视频帧并对其邻近视频帧进行采样，有效提高所选取视频帧的时空区分能力。借鉴图像分类中的随机数据增强方法，以随机数据增强方式对采样后视频短片段进行数据增强处理，使视频识别深度模型学习到更复杂的空间变化信息。根据2个公开的视频识别数据集和2个代表性的网络模型的评估实验结果表明，所提预处理方法可以使基准模型获得2.5个百分点以上的准确率提升，最高可提升6.8个百分点。上述实验结果验证了所提预处理方法在视频行为识别任务中的有效性。

关键词: 视频行为识别, 预处理方法, 动作指导的片段化视频采样, 数据增强, 深度学习

Abstract:

Video preprocessing operations, mainly including video frame sampling and data augmentation, are essential methods to improving the performance of deep video models in action recognition, and they have recently received increased attention. In this study, a novel data preprocessing method for video action recognition depth models is proposed, with a focus on video preprocessing problems of insufficient guidance during key frame sampling and relatively simple data amplification methods. First, a novel motion-guided fragmented video sampling technique is designed, which comprehensively considers features among different video frames and short-term timing features of video clips. It acquires key video frames guided by significant motion actions and sampling adjacent video frames of these key frames, which effectively improves the spatiotemporal discrimination capacity of the selected video frames. Additionally, motivated by the random data augmentation successfully applied in image classification task, this study further introduces the random data augmentation strategy to augment the sampled short video clips. This ensures that video recognition depth models can learn more complex spatially varying information. Based on evaluation experiments using two public video recognition datasets and two representative network models, the results show that the proposed preprocessing method can improve the accuracy of the baseline model by more than 2.5 percentage points, with the highest improvement accuracy of 6.8 percentage points. The results demonstrate the effectiveness of the method in video action recognition.

Key words: video action recognition, preprocessing method, motion-guided fragmented video sampling, data augmentation, deep learning

安峰民, 张冰冰, 董微, 张建新. 面向视频行为识别深度模型的数据预处理方法[J]. 计算机工程, 2024, 50(2): 281-287.

Fengmin AN, Bingbing ZHANG, Wei DONG, Jianxin ZHANG. Data Preprocessing Method for Video Action Recognition Depth Models[J]. Computer Engineering, 2024, 50(2): 281-287.

http://www.ecice06.com/CN/Y2024/V50/I2/281

图/表 9

图1 视频行为识别中的数据预处理过程

Fig.1 Data preprocessing process in video action recognition

图2 视频数据预处理过程

Fig.2 Process of video data preprocessing

图3 引入

$ \mathit{\mu } $

的运动累积曲线图

Fig.3 Motion accumulation curve with

$ \mathit{\mu } $

图4 2个数据集的部分图像帧示例

Fig.4 Partial image frame examples from two datasets

参考文献 29

1	HERATH S, HARANDI M, PORIKLI F. Going deeper into action recognition: a survey. Image and Vision Computing, 2017, 60, 4- 21. doi: 10.1016/j.imavis.2017.01.010
2	吴建超, 王利民, 武港山. 视频群体行为识别综述. 软件学报, 2023, 34(2): 964- 984.
	WU J C, WANG L M, WU G S. Group activity recognition in videos: a survey. Journal of Software, 2023, 34(2): 964- 984.
3	鹿天然, 于凤芹, 陈莹. 有效视频帧时间序池化的人体行为识别算法. 计算机工程, 2018, 44(12): 271-275, 287. doi: 10.19678/j.issn.1000-3428.0049178
	LU T R, YU F Q, CHEN Y. Human action recognition algorithm with temporal rank pooling of valid video frames. Computer Engineering, 2018, 44(12): 271-275, 287. doi: 10.19678/j.issn.1000-3428.0049178
4	孙彦玺, 赵婉婉, 武东辉, 等. 基于卷积长短时记忆网络的人体行为识别研究. 计算机工程, 2021, 47(10): 260- 268. doi: 10.19678/j.issn.1000-3428.0060938
	SUN Y X, ZHAO W W, WU D H, et al. Research of human activity recognition based on convolutional long short-term memory network. Computer Engineering, 2021, 47(10): 260- 268. doi: 10.19678/j.issn.1000-3428.0060938
5	PIERGIOVANNI A J, ANGELOVA A, RYOO M S. Tiny video networks. Applied AI Letters, 2022, 3(1): 38. doi: 10.1002/ail2.38
6	WANG L M, XIONG Y J, WANG Z, et al. Temporal segment networks: towards good practices for deep action recognition[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 20-36.
7	ZHI Y A, TONG Z, WANG L M, et al. MGSampler: an explainable sampling strategy for video action recognition[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 1513-1522.
8	FAN H H, XU Z W, ZHU L C, et al. Watching a small portion could be as good as watching all: towards efficient video classification[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence. New York, USA: ACM Press, 2018: 705-711.
9	CUBUK E D, ZOPH B, SHLENS J, et al. RandAugment: practical automated data augmentation with a reduced search space[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2020: 18613-18624.
10	YUN S, HAN D, CHUN S, et al. Cutmix: regularization strategy to train strong classifiers with localizable features[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 6023-6032.
11	YOO J, AHN N, SOHN K A. Rethinking data augmentation for image super-resolution: a comprehensive analysis and a new strategy[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 8375-8384.
12	MENG Y, LIN C C, PANDA R, et al. AR-Net: adaptive frame resolution for efficient action recognition[C]// Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 86-104.
13	KARPATHY A, TODERICI G, SHETTY S, et al. Large-scale video classification with convolutional neural networks[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2014: 1725-1732.
14	ZHU X Q, XU C, HUI L W, et al. Approximated bilinear modules for temporal modeling[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 3494-3503.
15	WALAWALKAR D, SHEN Z, LIU Z, et al. Attentive Cutmix: an enhanced data augmentation approach for deep learning based image classification[EB/OL]. [2023-12-11]. https://arxiv.org/abs/2003.13048.
16	CUBUK E D, ZOPH B, MANE D, et al. AutoAugment: learning augmentation strategies from data[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 113-123.
17	LIM S, KIM I, KIM T, et al. Fast AutoAugment[C]//Proceedings of the 33rd International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2019: 6665-6675.
18	GOYAL R, KAHOU S E, MICHALSKI V, et al. The "Something Something" video database for learning and evaluating visual common sense[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 5842-5850.
19	LIN J, GAN C A, HAN S. TSM: temporal shift module for efficient video understanding[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 7083-7093.
20	LI Y, JI B, SHI X T, et al. TEA: temporal excitation and aggregation for action recognition[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 909-918.
21	LUO C X, YUILLE A. Grouped spatial-temporal aggregation for efficient action recognition[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 5512-5521.
22	LIU Z, LUO D, WANG Y, et al. TEINet: towards an efficient architecture for video recognition[C]//Proceedings of AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2020: 11669-11676.
23	SUDHAKARAN S, ESCALERA S, LANZ O. Gate-shift networks for video action recognition[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 1102-1111.
24	WENG J W, LUO D H, WANG Y B, et al. Temporal distinct representation learning for action recognition[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 363-378.
25	KWON H, KIM M, KWAK S, et al. MotionSqueeze: neural motion feature learning for video understanding[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 345-362.
26	WU W H, HE D L, LIN T W, et al. MVFNet: multi-view fusion network for efficient video recognition[C]//Proceedings of AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2021: 2943-2951.
27	SU H, FENG J, WANG D, et al. TSI: temporal saliency integration for video action recognition[EB/OL]. [2023-12-11]. https://arxiv.org/abs/2106.01088.
28	WANG L M, TONG Z, JI B, et al. TDN: temporal difference networks for efficient action recognition[C]//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 1895-1904.
29	CHEN X, HAN Y, WANG X, et al. Action keypoint network for efficient video recognition. IEEE Transactions on Image Processing, 2022, 31, 4980- 4993. doi: 10.1109/TIP.2022.3191461

[1]	徐浩宸, 刘满华. 基于多层次自注意力网络的人脸特征点检测[J]. 计算机工程, 2024, 50(2): 239-246.
[2]	曾嘉忻, 张卫明, 张荣. 基于后门的鲁棒后向模型水印方法[J]. 计算机工程, 2024, 50(2): 132-139.
[3]	陈虹, 王瀚文, 金海波. 融合改进自编码器和残差网络的入侵检测模型[J]. 计算机工程, 2024, 50(2): 188-195.
[4]	郑晨俊, 曾艳, 袁俊峰, 张纪林, 王鑫, 韩猛. 基于联邦学习的船舶AIS轨迹预测算法[J]. 计算机工程, 2024, 50(2): 298-307.
[5]	丁国辉, 刘宇琪, 王言开, 耿施展, 姜天昊. 基于翻转网络的低相关性序列数据预测研究[J]. 计算机工程, 2024, 50(2): 78-90.
[6]	圣文顺, 余熊峰, 林佳燕, 陈欣. 融合注意力与特征金字塔的小尺度目标检测算法[J]. 计算机工程, 2024, 50(1): 242-250.
[7]	祝冰艳, 陈志华, 盛斌. 基于感知增强Swin Transformer的遥感图像检测[J]. 计算机工程, 2024, 50(1): 216-223.
[8]	蒋心璐, 陈天恩, 王聪, 赵春江. 大田环境下的农业害虫图像小目标检测算法[J]. 计算机工程, 2024, 50(1): 232-241.
[9]	白尚旺, 王梦瑶, 胡静, 陈志泊. 多区域注意力的细粒度图像分类网络[J]. 计算机工程, 2024, 50(1): 271-278.
[10]	曹广硕, 黄瑞章, 陈艳平, 秦永彬. 基于多模态学习的乳腺癌生存预测研究[J]. 计算机工程, 2024, 50(1): 296-305.
[11]	池亚平, 岳梓岩, 林雨衡. 基于Transformer的SM4算法工作模式识别[J]. 计算机工程, 2023, 49(9): 109-117.
[12]	林中霖, 时金桥, 王美琪, 王学宾, 王雨燕. 基于应用行为划分的Android恶意应用检测技术[J]. 计算机工程, 2023, 49(9): 125-136.
[13]	江雨燕, 陶承凤, 李平. 数据增强和自适应自步学习的深度子空间聚类算法[J]. 计算机工程, 2023, 49(8): 96-103, 110.
[14]	李泽水, 冀俊忠, 杨翠翠. 基于边权重信息深度网络嵌入的PPIN功能模块检测[J]. 计算机工程, 2023, 49(8): 69-76.
[15]	王可铮, 徐玉芬, 周尚波. 结合对比感知损失和融合注意力的图像去雾模型[J]. 计算机工程, 2023, 49(8): 207-214.

选择文件类型/文献管理软件名称

选择包含的内容