基于自适应多尺度图卷积网络的骨架动作识别

doi:10.19678/j.issn.1000-3428.0065882

摘要/Abstract

摘要：

将人体骨架建模为时空拓扑图的图卷积网络在基于人体骨架数据的动作识别任务中得到了广泛应用。但现有图卷积网络存在预定义骨架拓扑图拓扑结构固定、单支路时间图卷积算子提取时空特征粒度单一的问题，极大限制了模型的泛化能力和表达能力。提出基于自适应多尺度图卷积网络的人体骨架动作识别模型，自适应空间图卷积层将骨架的拓扑结构作为参数进行端到端的自适应学习，根据动作生成数据驱动的骨架拓扑图。多尺度时间图卷积层对时间图卷积算子进行多支路扩展，动态融合骨架序列不同时间粒度的时空特征。综合骨架关节点、骨架长度、骨架关节点运动、骨架长度运动4路信息输入模型。实验结果表明，所提模型在NTU RGB+D 60动作识别数据集下的人物划分(CS)模式和视角划分模式实验中分别取得90.5%和96.8%的识别准确率，在NTU RGB+D 120动作识别数据集的CS模式和设置划分模式的实验中分别取得86.0%和88.7%的识别准确率，能有效提取骨架动作的时空特征，提升了人体骨架动作识别的分类性能。

关键词: 人体骨架, 动作识别, 图卷积网络, 自适应, 多尺度

Abstract:

The Graph Convolution Network(GCN) is widely used for skeleton-based recognition methods, whereby a predefined spatiotemporal topology graph is used to model human skeletal features. However, the existing GCN has limitations in two aspects: the predefined skeleton topology structure limits the generalization ability of the model, and single-granular spatiotemporal features limit the capacity of the model. To solve the aforementioned problems, an adaptive multi-scale graph convolution network is proposed. The adaptive spatial graph convolution layer regards the topology of the skeleton as parameters to optimize, thereby generating data-driven skeleton topology by samples. The multiscale temporal graph convolution layer uses various temporal graph convolution kernels to dynamically integrate the multi-granular spatiotemporal features. Extensive experiments were conducted by combining joint, bone, joint motion, and bone motion streams as inputs of the proposed model. The experimental results show that the classification accuracy of the proposed model under NTU RGB+D 60 action recognition data set for the Cross-Subject(CS) and Cross-View(CV) subsets was 90.5% and 96.8%, respectively, and the classification accuracy of the NTU RGB+D 120 action recognition dataset for the CS and Cross-Setup(CT)subsets was 86.0% and 88.7%, respectively. The model can effectively extract the spatiotemporal features of skeleton motion, thereby improving the classification performance of human skeleton motion recognition.

Key words: human skeleton, action recognition, Graph Convolution Network(GCN), adaption, multi-scale

刘宽, 奚小冰, 周明东. 基于自适应多尺度图卷积网络的骨架动作识别[J]. 计算机工程, 2023, 49(10): 264-271.

Kuan LIU, Xiaobing XI, Mingdong ZHOU. Skeleton Action Recognition Based on Adaptive Multi-scale Graph Convolutional Network[J]. Computer Engineering, 2023, 49(10): 264-271.

https://www.ecice06.com/CN/Y2023/V49/I10/264

图/表 11

图1 空间图卷积算子和时间图卷积算子

Fig.1 Spatial graph convolution operator and temporal graph convolution operator

图2 自适应空间图卷积层的流程

Fig.2 Procedure of adaptive spatial graph convolution network

图3 k阶邻接矩阵

Fig.3 k-hop adjacency matrix

图4 多尺度时间图卷积层的算法流程

Fig.4 Algorithm procedure of multi-scale temporal graph convolution network

图5 时空图卷积网络模型的整体架构

Fig.5 Overall architecture of spatiotemporal graph convolutional network model

图6 原始骨架序列和预处理后骨架序列对比

Fig.6 Comparison between original skeleton sample and preprocessed skeleton sample

图7 预定义骨架拓扑图和自适应骨架拓扑图对比

Fig.7 Comparison between predefined skeleton topology graph and adaptive skeleton topology graph

参考文献 40

1	WANG X, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]// Proceedings of IEEE Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7794-7803.
2	CARREIRA J, ZISSERMAN A. Quo vadis, action recognition? A new model and the kinetics dataset[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 4724-4733.
3	WANG L, HUYNH D Q, KONIUSZ P. A comparative review of recent kinect-based action recognition algorithms. IEEE Transactions on Image Processing, 2020, 29, 15- 28. doi: 10.1109/TIP.2019.2925285
4	胡建芳, 王熊辉, 郑伟诗, 等. RGB-D行为识别研究进展及展望. 自动化学报, 2019, 45 (5): 829- 840. URL
	HU J F, WANG X H, ZHENG W S, et al. RGB-D action recognition: recent advances and future perspectives. Acta Automatica Sinica, 2019, 45 (5): 829- 840. URL
5	赫磊, 邵展鹏, 张剑华, 等. 基于深度学习的行为识别算法综述. 计算机科学, 2020, 47 (S01): 139- 147. doi: 10.11896/jsjkx.190900176
	HE L, SHAO Z P, ZHANG J H, et al. Review of deep learning-based action recognition algorithms. Computer Science, 2020, 47 (S01): 139- 147. doi: 10.11896/jsjkx.190900176
6	ZHANG Z Y. Microsoft kinect sensor and its effect. IEEE MultiMedia, 2012, 19 (2): 4- 10. doi: 10.1109/MMUL.2012.24
7	VEMULAPALLI R, ARRATE F, CHELLAPPA R. Human action recognition by representing 3D skeletons as points in a lie group[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2014: 588-595.
8	KONIUSZ P, CHERIAN A, PORIKLI F. Tensor representations via kernel linearization for action recognition from 3D skeletons[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 37-53.
9	KONIUS Z P, WANG L, CHERIAN A. Tensor representations for action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (2): 648- 665. doi: 10.1109/TPAMI.2021.3107160
10	HU J F, ZHENG W S, LAI J H, et al. Jointly learning heterogeneous features for RGB-D activity recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 5344-5352.
11	SHAHROUDY A, LIU J, NG T T, et al. NTU RGB D: a large scale dataset for 3D human activity analysis[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 1010-1019.
12	LI S, LI W Q, COOK C, et al. Independently recurrent neural network: building a longer and deeper RNN[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 5457-5466.
13	SI C Y, JING Y, WANG W, et al. Skeleton-based action recognition with spatial reasoning and temporal stack learning[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 106-121.
14	DU Y, WANG W, WANG L. Hierarchical recurrent neural network for skeleton based action recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 1110-1118.
15	LI W B, WEN L Y, CHANG M C, et al. Adaptive RNN tree for large-scale human action recognition[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 1453-1461.
16	LIU J, SHAHROUDY A, XU D, et al. Spatio-temporal LSTM with trust gates for 3D human action recognition[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 816-833.
17	LIU J, WANG G, DUAN L Y, et al. Skeleton-based human action recognition with global context-aware attention LSTM networks. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2018, 27 (4): 1586- 1599. doi: 10.1109/TIP.2017.2785279
18	KE Q H, BENNAMOUN M, AN S J, et al. A new representation of skeleton sequences for 3D action recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 4570-4579.
19	KIM T S, REITER A. Interpretable 3D human action analysis with temporal convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington D. C., USA: IEEE Press, 2017: 1623-1631.
20	LIU H, TU J, LIU M. Two-stream 3D convolutional neural network for skeleton-based action recognition[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1705.08106.
21	LIU M, LIU H, CHEN C. Enhanced skeleton visualization for view invariant human action recognition. Pattern Recognition, 2017, 68, 346- 362. doi: 10.1016/j.patcog.2017.02.030
22	LIU M Y, YUAN J S. Recognizing human actions as the evolution of pose estimation maps[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 1159-1168.
23	YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32 (1): 7444- 7452. doi: 10.48550/arXiv.1801.07455
24	SHI L, ZHANG Y F, CHENG J, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 12018-12027.
25	SHI L, ZHANG Y F, CHENG J, et al. Skeleton-based action recognition with multi-stream adaptive graph convolutional networks. IEEE Transactions on Image Processing, 2020, 29, 9532- 9545. doi: 10.1109/TIP.2020.3028207
26	SHI L, ZHANG Y F, CHENG J, et al. Skeleton-based action recognition with directed graph neural networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 7904-7913.
27	LIU Z Y, ZHANG H W, CHEN Z H, et al. Disentangling and unifying graph convolutions for skeleton-based action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 140-149.
28	DING X L, YANG K, CHEN W. A semantics-guided graph convolutional network for skeleton-based action recognition[C]//Proceedings of the 4th International Conference on Innovation in Artificial Intelligence. New York, USA: ACM Press, 2020: 130-136.
29	孙琪翔, 何宁, 张聪聪, 等. 基于轻量级图卷积的人体骨架动作识别方法. 计算机工程, 2022, 48 (5): 306- 313. URL
	SUN Q X, HE N, ZHANG C C, et al. Human skeleton action recognition method based on lightweight graph convolution. Computer Engineering, 2022, 48 (5): 306- 313. URL
30	CHEN Y X, ZHANG Z Q, YUAN C F, et al. Channel-wise topology refinement graph convolution for skeleton-based action recognition[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2022: 13339-13348.
31	王小娟, 钟云, 金磊, 等. 基于骨架的自适应尺度图卷积动作识别. 天津大学学报(自然科学与工程技术版), 2022, 55 (3): 306- 312. URL
	WANG X J, ZHONG Y, JIN L, et al. Scale adaptive graph convolutional network for skeleton-based action recognition. Journal of Tianjin University (Science and Technology), 2022, 55 (3): 306- 312. URL
32	ZHANG J X, YE G X, TU Z G, et al. A spatial attentive and temporal dilated (SATD) GCN for skeleton-based action recognition. CAAI Transactions on Intelligence Technology, 2022, 7 (1): 46- 55. doi: 10.1049/cit2.12012
33	TU Z G, ZHANG J X, LI H Y, et al. Joint-bone fusion graph convolutional network for semi-supervised skeleton action recognition. IEEE Transactions on Multimedia, 2023, 25, 1819- 1831. doi: 10.48550/arXiv.2202.04075
34	BIAN C L, FENG W, WANG S. Self-supervised representation learning for skeleton-based group activity recognition[C]//Proceedings of the 30th ACM International Conference on Multimedia. New York, USA: ACM Press, 2022: 5990-5998.
35	BRUNA J, ZAREMBA W, SZLAM A, et al. Spectral networks and locally connected networks on graphs[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1312.6203.
36	KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1609.02907.
37	NIEPERT M, AHMED M, KUTZKOV K. Learning convolutional neural networks for graphs[C]// Proceedings of the 33rd International Conference on Machine Learning. New York, USA: ACM Press, 2016: 2014-2023.
38	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of Conference on Advances in Neural Information. Washington D. C., USA: IEEE Press, 2017: 76-85.
39	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washing D. C., USA: IEEE Press, 2016: 770-778.
40	LIU J, SHAHROUDY A, PEREZ M, et al. NTU RGB+D 120: a large-scale benchmark for 3D human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (10): 2684- 2701. doi: 10.1109/tpami.2019.2916873

[1]	曾碧卿, 陈鹏飞, 姚勇涛. 融合思维链和低秩自适应微调的方面情感三元组抽取[J]. 计算机工程, 2024, 50(7): 53-62.
[2]	李云航, 潘晴, 田妮莉. 结构相似度优化的混合多尺度医学图像融合[J]. 计算机工程, 2024, 50(7): 264-270.
[3]	杨郅树, 梁佳楠, 曹永军, 钟震宇, 何永伦. 基于局部分离与多尺度融合的图像超分辨率重建[J]. 计算机工程, 2024, 50(7): 314-323.
[4]	林芷薇, 杨祖元, 王斯秋, 杨超. 基于多尺度线性全局注意力的运动员检测算法[J]. 计算机工程, 2024, 50(7): 352-359.
[5]	秦媛, 张杭, 朱宏鹏, 李炯, 胡航. 卫星MIMO通信系统抗移动无人机集群干扰的在线BSS算法[J]. 计算机工程, 2024, 50(6): 65-76.
[6]	胡庆. 多尺度融合与双输出U-Net网络的行人重识别[J]. 计算机工程, 2024, 50(6): 102-109.
[7]	更藏措毛, 黄鹤鸣, 杨毅杰. 融合多尺度特征与上下文信息的语音增强方法[J]. 计算机工程, 2024, 50(6): 138-147.
[8]	于洋, 孙芳芳, 吕华, 李扬, 王晓民. 基于多尺度时空注意力网络的微表情检测方法[J]. 计算机工程, 2024, 50(6): 228-235.
[9]	杨硕, 王一丁. 基于改进薄板样条运动模型的人脸动画算法[J]. 计算机工程, 2024, 50(6): 255-265.
[10]	代巍, 王丰羽, 冀常鹏. 基于情感增强与双图卷积网络的方面级情感分析[J]. 计算机工程, 2024, 50(5): 120-127.
[11]	孙帮勇, 马铭, 于涛. 基于区域特征强化的多尺度伪装目标检测方法[J]. 计算机工程, 2024, 50(5): 209-219.
[12]	李田芳, 普园媛, 赵征鹏, 徐丹, 钱文华. 基于CLIP和双空间自适应归一化的图像翻译[J]. 计算机工程, 2024, 50(5): 229-240.
[13]	李晶, 李健, 陈海丰, 张倩, 王丽燕, 裴二成. 基于关键区域遮挡与重建的人脸表情识别[J]. 计算机工程, 2024, 50(5): 241-249.
[14]	黄君泽, 吴文渊, 李轶, 石明全, 王正江. 面向动态公交的离散分层记忆粒子群优化算法[J]. 计算机工程, 2024, 50(4): 20-30.
[15]	莫少聪, 陈庆锋, 谢泽, 刘春雨, 邱俊铼. 基于动态图注意力与标签传播的实体对齐[J]. 计算机工程, 2024, 50(4): 150-159.

选择文件类型/文献管理软件名称

选择包含的内容