基于轻量级图卷积的人体骨架动作识别方法

doi:10.19678/j.issn.1000-3428.0061304

摘要/Abstract

摘要： 视频中的人体动作识别在计算机视觉领域得到广泛关注，基于人体骨架的动作识别方法可以明确地表现人体动作，因此已逐渐成为该领域的重要研究方向之一。针对多数主流人体动作识别方法网络参数量大、计算复杂度高等问题，设计一种融合多流数据的轻量级图卷积网络，并将其应用于人体骨架动作识别任务。在数据预处理阶段，利用多流数据融合方法对4种特征数据流进行融合，通过一次训练就可得到最优结果，从而降低网络参数量。设计基于图卷积网络的非局部网络模块，以捕获图像的全局信息从而提高动作识别准确率。在此基础上，设计空间Ghost图卷积模块和时间Ghost图卷积模块，从网络结构上进一步降低网络参数量。在动作识别数据集NTU60 RGB+D和NTU120 RGB+D上进行实验，结果表明，与近年主流动作识别方法ST-GCN、2s AS-GCN、2s AGCN等相比，基于该轻量级图卷积网络的人体骨架动作识别方法在保持较低网络参数量的情况下能够取得较高的识别准确率。

关键词: 人体骨架动作识别, 数据融合, 图卷积, 非局部网络模块, Ghost网络

Abstract: Human action recognition in video has garnered extensive attention in the field of computer vision.The action recognition method based on human skeleton can clearly represent human motion;therefore, it has gradually become one of the most important research directions in the abovementioned field.To solve the issue of numerous network parameters and high computational complexity in most mainstream human action recognition methods, a lightweight graph convolution network integrating multistream data is designed and applied to human skeleton action recognition.In the data preprocessing stage, the multistream data fusion method is used to fuse four characteristic data streams.Optimal results can be obtained via one round of training;as such, the number of network parameters required is reduced.A non-local network module based on graph convolution network is designed to capture the global information of an image to improve the accuracy of action recognition.Subsequently, a space Ghost graph convolution module and a time Ghost graph convolution module are designed to further reduce the number of network parameters from the network structure.Experiments are performed on action recognition datasets NTU60 RGB+D and NTU120 RGB+D.Results show that compared with recent mainstream action recognition methods ST-GCN, 2s AS-GCN, and 2s AGCN, the human skeleton action recognition method based on the lightweight graph convolution network can achieve higher recognition accuracy while maintaining a lower number of network parameters.

Key words: human skeleton action recognition, data fusion, graph convolution, non-local network module, Ghost network

中图分类号:

TP391.4

孙琪翔, 何宁, 张聪聪, 刘圣杰. 基于轻量级图卷积的人体骨架动作识别方法[J]. 计算机工程, 2022, 48(5): 306-313.

SUN Qixiang, HE Ning, ZHANG Congcong, LIU Shengjie. Human Skeleton Action Recognition Method Based on Lightweight Graph Convolution[J]. Computer Engineering, 2022, 48(5): 306-313.

https://www.ecice06.com/CN/Y2022/V48/I5/306

图/表 12

20220806175548

20220806175551

20220806175555

20220806175559

20220806175602

20220806175606

20220806175609

20220806175613

20220806175616

20220806175620

20220806175623

20220806175627

参考文献

[1] 胡建芳, 王熊辉, 郑伟诗, 等.RGB-D行为识别研究进展及展望[J].自动化学报, 2019, 45(5):829-840. HU J F, WANG X H, ZHENG W S, et al.RGB-D action recognition:recent advances and future perspectives[J].Acta Automatica Sinica, 2019, 45(5):829-840.(in Chinese)
[2] 赫磊, 邵展鹏, 张剑华, 等.基于深度学习的行为识别算法综述[J].计算机科学, 2020, 47(S1):139-147. HE L, SHAO Z P, ZHANG J H, et al.Review of deep learning-based action recognition algorithms[J].Computer Science, 2020, 47(S1):139-147.(in Chinese)
[3] 张聪聪, 何宁, 孙琪翔, 等.基于注意力机制的3D DenseNet人体动作识别方法[J].计算机工程, 2021, 47(11):313-320. ZHANG C C, HE N, SUN Q X, et al.Human motion recognition method based on attention mechanism of 3D DenseNet[J].Computer Engineering, 2021, 47(11):313-320.(in Chinese)
[4] ZHANG Z M, MA X, SONG R, et al.Deep learning based human action recognition:a survey[C]//Proceedings of Chinese Automation Congress.Jinan, China:[s.n.], 2017:3780-3785.
[5] YANG X D, TIAN Y L.EigenJoints-based action recognition using naïve-Bayes-nearest-neighbor[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.Washington D.C., USA:IEEE Press, 2012:14-19.
[6] LECUN Y, BOTTOU L, BENGIO Y, et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE, 1998, 86(11):2278-2324.
[7] RUMELHART D E, HINTON G E, WILLIAMS R J.Learning representations by back-propagating errors[J].Nature, 1986, 323(6088):533-536.
[8] KIPF T, FETAYA E, WANG K C, et al.Neural relational inference for interacting systems[EB/OL].[2021-02-05].https://arxiv.org/abs/1802.04687.
[9] LIU H, TU J H, LIU M Y.Two-stream 3D convolutional neural network for skeleton-based action recognition[EB/OL].[2021-02-05].https://arxiv.org/abs/1705.08106.
[10] DU Y, WANG W, WANG L.Hierarchical recurrent neural network for skeleton based action recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:1110-1118.
[11] WANG X L, GIRSHICK R, GUPTA A, et al.Non-local neural networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:7794-7803.
[12] HAN K, WANG Y H, TIAN Q, et al.GhostNet:more features from cheap operations[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2020:1577-1586.
[13] YAN S J, XIONG Y J, LIN D H.Spatial temporal graph convolutional networks for skeleton-based action recognition[EB/OL].[2021-02-05].https://arxiv.org/abs/1801.07455.
[14] SHI L, ZHANG Y F, CHENG J, et al.Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2019:12018-12027.
[15] LI M S, CHEN S H, CHEN X, et al.Actional-structural graph convolutional networks for skeleton-based action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2019:3590-3598.
[16] SI C Y, CHEN W T, WANG W, et al.An attention enhanced graph convolutional LSTM network for skeleton-based action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2019:1227-1236.
[17] SHI L, ZHANG Y F, CHENG J, et al.Skeleton-based action recognition with directed graph neural networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2019:7904-7913.
[18] YAN S J, XIONG Y J, LIN D H.Spatial temporal graph convolutional networks for skeleton-based action recognition[EB/OL].[2021-02-05].https://arxiv.org/abs/1801.07455.
[19] 孔玮, 刘云, 李辉, 等.基于图卷积网络的行为识别方法综述[J].控制与决策, 2021, 36(7):1537-1546. KONG W, LIU Y, LI H, et al.A survey of action recognition methods based on graph convolutional network[J].Control and Decision, 2021, 36(7):1537-1546.(in Chinese)
[20] SHI L, ZHANG Y F, CHENG J, et al.Decoupled spatial-temporal attention network for skeleton-based action-gesture recognition[EB/OL].[2021-02-05].https://openaccess.thecvf.com/content/ACCV2020/papers/Shi_Decoupled_Spatial-Temporal_Attention_Network_for_Skeleton-Based_Action-Gesture_Recognition_ACCV_2020_paper.pdf.
[21] ZOPH B, VASUDEVAN V, SHLENS J, et al.Learning transferable architectures for scalable image recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2018:8697-8710.
[22] VANHOUCKE V, SENIOR A, MAO M Z.Improving the speed of neural networks on CPUs[EB/OL].[2021-02-05].https://research.google.com/pubs/archive/37631.pdf.
[23] GONG Y C, LIU L, YANG M, et al.Compressing deep convolutional networks using vector quantization[EB/OL].[2021-02-05].https://arxiv.org/pdf/1412.6115.pdf.
[24] COURBARIAUX M, BENGIO Y, DAVID J P.BinaryConnect:training deep neural networks with binary weights during propagations[EB/OL].[2021-02-05].https://arxiv.org/abs/1511.00363.
[25] COURBARIAUX M, HUBARA I, SOUDRY D, et al.Binarized neural networks:training deep neural networks with weights and activations constrained to +1 or -1[EB/OL].[2021-02-05].https://arxiv.org/pdf/1602.02830.pdf.
[26] RASTEGARI M, ORDONEZ V, REDMON J, et al.XNOR-Net:ImageNet classification using binary convolutional neural networks[EB/OL].[2021-02-05].https://arxiv.org/pdf/1603.05279.pdf.
[27] HANSON S J, PRATT L Y.Comparing biases for minimal network construction with back-propagation[EB/OL].[2021-02-05].https://proceedings.neurips.cc/paper/1988/file/1c9ac0159c94d8d0cbedc973445af2da-Paper.pdf.
[28] IANDOLA F N, HAN S, MOSKEWICZ M W, et al.SqueezeNet:AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size[EB/OL].[2021-02-05].https://arxiv.org/abs/1602.07360.
[29] HOWARD A G, ZHU M L, CHEN B, et al.MobileNets:efficient convolutional neural networks for mobile vision applications[EB/OL].[2021-02-05].https://arxiv.org/abs/1704.04861.
[30] 张洋, 姚登峰, 江铭虎, 等.基于EfficientDet网络的细粒度吸烟行为识别[J].计算机工程, 2022, 48(3):302-309, 314. ZHANG Y, YAO D F, JIANG M H, et al.Fine-grained smoking behavior recognition based on EfficientDet network[J].Computer Engineering, 2022, 48(3):302-309, 314.(in Chinese)
[31] SHAHROUDY A, LIU J, NG T T, et al.NTU RGB+D:a large scale dataset for 3D human activity analysis[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:1010-1019.
[32] LIU J, SHAHROUDY A, PEREZ M, et al.NTU RGB D 120:a large-scale benchmark for 3D human activity understanding[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(10):2684-2701.
[33] SHUMAN D I, NARANG S K, FROSSARD P, et al.The emerging field of signal processing on graphs:extending high-dimensional data analysis to networks and other irregular domains[J].IEEE Signal Processing Magazine, 2013, 30(3):83-98.
[34] LI B, LI X, ZHANG Z F, et al.Spatio-temporal graph routing for skeleton-based action recognition[J].Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33:8561-8568.
[35] PASZKE A, GROSS S, CHINTALA S, et al.Automatic differentiation in PyTorch[EB/OL].[2021-02-05].https://openreview.net/pdf?id=BJJsrmfCZ.

选择文件类型/文献管理软件名称

选择包含的内容