基于自适应多尺度图卷积网络的骨架动作识别

doi:10.19678/j.issn.1000-3428.0065882

摘要/Abstract

摘要：

将人体骨架建模为时空拓扑图的图卷积网络在基于人体骨架数据的动作识别任务中得到了广泛应用。但现有图卷积网络存在预定义骨架拓扑图拓扑结构固定、单支路时间图卷积算子提取时空特征粒度单一的问题，极大限制了模型的泛化能力和表达能力。提出基于自适应多尺度图卷积网络的人体骨架动作识别模型，自适应空间图卷积层将骨架的拓扑结构作为参数进行端到端的自适应学习，根据动作生成数据驱动的骨架拓扑图。多尺度时间图卷积层对时间图卷积算子进行多支路扩展，动态融合骨架序列不同时间粒度的时空特征。综合骨架关节点、骨架长度、骨架关节点运动、骨架长度运动4路信息输入模型。实验结果表明，所提模型在NTU RGB+D 60动作识别数据集下的人物划分(CS)模式和视角划分模式实验中分别取得90.5%和96.8%的识别准确率，在NTU RGB+D 120动作识别数据集的CS模式和设置划分模式的实验中分别取得86.0%和88.7%的识别准确率，能有效提取骨架动作的时空特征，提升了人体骨架动作识别的分类性能。

关键词: 人体骨架, 动作识别, 图卷积网络, 自适应, 多尺度

Abstract:

The Graph Convolution Network(GCN) is widely used for skeleton-based recognition methods, whereby a predefined spatiotemporal topology graph is used to model human skeletal features. However, the existing GCN has limitations in two aspects: the predefined skeleton topology structure limits the generalization ability of the model, and single-granular spatiotemporal features limit the capacity of the model. To solve the aforementioned problems, an adaptive multi-scale graph convolution network is proposed. The adaptive spatial graph convolution layer regards the topology of the skeleton as parameters to optimize, thereby generating data-driven skeleton topology by samples. The multiscale temporal graph convolution layer uses various temporal graph convolution kernels to dynamically integrate the multi-granular spatiotemporal features. Extensive experiments were conducted by combining joint, bone, joint motion, and bone motion streams as inputs of the proposed model. The experimental results show that the classification accuracy of the proposed model under NTU RGB+D 60 action recognition data set for the Cross-Subject(CS) and Cross-View(CV) subsets was 90.5% and 96.8%, respectively, and the classification accuracy of the NTU RGB+D 120 action recognition dataset for the CS and Cross-Setup(CT)subsets was 86.0% and 88.7%, respectively. The model can effectively extract the spatiotemporal features of skeleton motion, thereby improving the classification performance of human skeleton motion recognition.

Key words: human skeleton, action recognition, Graph Convolution Network(GCN), adaption, multi-scale

刘宽, 奚小冰, 周明东. 基于自适应多尺度图卷积网络的骨架动作识别[J]. 计算机工程, 2023, 49(10): 264-271.

Kuan LIU, Xiaobing XI, Mingdong ZHOU. Skeleton Action Recognition Based on Adaptive Multi-scale Graph Convolutional Network[J]. Computer Engineering, 2023, 49(10): 264-271.

http://www.ecice06.com/CN/Y2023/V49/I10/264

图/表 11

图1 空间图卷积算子和时间图卷积算子

Fig.1 Spatial graph convolution operator and temporal graph convolution operator

图2 自适应空间图卷积层的流程

Fig.2 Procedure of adaptive spatial graph convolution network

图3 k阶邻接矩阵

Fig.3 k-hop adjacency matrix

图4 多尺度时间图卷积层的算法流程

Fig.4 Algorithm procedure of multi-scale temporal graph convolution network

图5 时空图卷积网络模型的整体架构

Fig.5 Overall architecture of spatiotemporal graph convolutional network model

图6 原始骨架序列和预处理后骨架序列对比

Fig.6 Comparison between original skeleton sample and preprocessed skeleton sample

图7 预定义骨架拓扑图和自适应骨架拓扑图对比

Fig.7 Comparison between predefined skeleton topology graph and adaptive skeleton topology graph

参考文献 40

1	WANG X, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]// Proceedings of IEEE Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7794-7803.
2	CARREIRA J, ZISSERMAN A. Quo vadis, action recognition? A new model and the kinetics dataset[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 4724-4733.
3	WANG L, HUYNH D Q, KONIUSZ P. A comparative review of recent kinect-based action recognition algorithms. IEEE Transactions on Image Processing, 2020, 29, 15- 28. doi: 10.1109/TIP.2019.2925285
4	胡建芳, 王熊辉, 郑伟诗, 等. RGB-D行为识别研究进展及展望. 自动化学报, 2019, 45 (5): 829- 840. URL
	HU J F, WANG X H, ZHENG W S, et al. RGB-D action recognition: recent advances and future perspectives. Acta Automatica Sinica, 2019, 45 (5): 829- 840. URL
5	赫磊, 邵展鹏, 张剑华, 等. 基于深度学习的行为识别算法综述. 计算机科学, 2020, 47 (S01): 139- 147. doi: 10.11896/jsjkx.190900176
	HE L, SHAO Z P, ZHANG J H, et al. Review of deep learning-based action recognition algorithms. Computer Science, 2020, 47 (S01): 139- 147. doi: 10.11896/jsjkx.190900176
6	ZHANG Z Y. Microsoft kinect sensor and its effect. IEEE MultiMedia, 2012, 19 (2): 4- 10. doi: 10.1109/MMUL.2012.24
7	VEMULAPALLI R, ARRATE F, CHELLAPPA R. Human action recognition by representing 3D skeletons as points in a lie group[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2014: 588-595.
8	KONIUSZ P, CHERIAN A, PORIKLI F. Tensor representations via kernel linearization for action recognition from 3D skeletons[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 37-53.
9	KONIUS Z P, WANG L, CHERIAN A. Tensor representations for action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (2): 648- 665. doi: 10.1109/TPAMI.2021.3107160
10	HU J F, ZHENG W S, LAI J H, et al. Jointly learning heterogeneous features for RGB-D activity recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 5344-5352.
11	SHAHROUDY A, LIU J, NG T T, et al. NTU RGB D: a large scale dataset for 3D human activity analysis[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 1010-1019.
12	LI S, LI W Q, COOK C, et al. Independently recurrent neural network: building a longer and deeper RNN[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 5457-5466.
13	SI C Y, JING Y, WANG W, et al. Skeleton-based action recognition with spatial reasoning and temporal stack learning[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 106-121.
14	DU Y, WANG W, WANG L. Hierarchical recurrent neural network for skeleton based action recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 1110-1118.
15	LI W B, WEN L Y, CHANG M C, et al. Adaptive RNN tree for large-scale human action recognition[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 1453-1461.
16	LIU J, SHAHROUDY A, XU D, et al. Spatio-temporal LSTM with trust gates for 3D human action recognition[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 816-833.
17	LIU J, WANG G, DUAN L Y, et al. Skeleton-based human action recognition with global context-aware attention LSTM networks. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2018, 27 (4): 1586- 1599. doi: 10.1109/TIP.2017.2785279
18	KE Q H, BENNAMOUN M, AN S J, et al. A new representation of skeleton sequences for 3D action recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 4570-4579.
19	KIM T S, REITER A. Interpretable 3D human action analysis with temporal convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition Workshops. Washington D. C., USA: IEEE Press, 2017: 1623-1631.
20	LIU H, TU J, LIU M. Two-stream 3D convolutional neural network for skeleton-based action recognition[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1705.08106.
21	LIU M, LIU H, CHEN C. Enhanced skeleton visualization for view invariant human action recognition. Pattern Recognition, 2017, 68, 346- 362. doi: 10.1016/j.patcog.2017.02.030
22	LIU M Y, YUAN J S. Recognizing human actions as the evolution of pose estimation maps[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 1159-1168.
23	YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32 (1): 7444- 7452. doi: 10.48550/arXiv.1801.07455
24	SHI L, ZHANG Y F, CHENG J, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 12018-12027.
25	SHI L, ZHANG Y F, CHENG J, et al. Skeleton-based action recognition with multi-stream adaptive graph convolutional networks. IEEE Transactions on Image Processing, 2020, 29, 9532- 9545. doi: 10.1109/TIP.2020.3028207
26	SHI L, ZHANG Y F, CHENG J, et al. Skeleton-based action recognition with directed graph neural networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 7904-7913.
27	LIU Z Y, ZHANG H W, CHEN Z H, et al. Disentangling and unifying graph convolutions for skeleton-based action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 140-149.
28	DING X L, YANG K, CHEN W. A semantics-guided graph convolutional network for skeleton-based action recognition[C]//Proceedings of the 4th International Conference on Innovation in Artificial Intelligence. New York, USA: ACM Press, 2020: 130-136.
29	孙琪翔, 何宁, 张聪聪, 等. 基于轻量级图卷积的人体骨架动作识别方法. 计算机工程, 2022, 48 (5): 306- 313. URL
	SUN Q X, HE N, ZHANG C C, et al. Human skeleton action recognition method based on lightweight graph convolution. Computer Engineering, 2022, 48 (5): 306- 313. URL
30	CHEN Y X, ZHANG Z Q, YUAN C F, et al. Channel-wise topology refinement graph convolution for skeleton-based action recognition[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2022: 13339-13348.
31	王小娟, 钟云, 金磊, 等. 基于骨架的自适应尺度图卷积动作识别. 天津大学学报(自然科学与工程技术版), 2022, 55 (3): 306- 312. URL
	WANG X J, ZHONG Y, JIN L, et al. Scale adaptive graph convolutional network for skeleton-based action recognition. Journal of Tianjin University (Science and Technology), 2022, 55 (3): 306- 312. URL
32	ZHANG J X, YE G X, TU Z G, et al. A spatial attentive and temporal dilated (SATD) GCN for skeleton-based action recognition. CAAI Transactions on Intelligence Technology, 2022, 7 (1): 46- 55. doi: 10.1049/cit2.12012
33	TU Z G, ZHANG J X, LI H Y, et al. Joint-bone fusion graph convolutional network for semi-supervised skeleton action recognition. IEEE Transactions on Multimedia, 2023, 25, 1819- 1831. doi: 10.48550/arXiv.2202.04075
34	BIAN C L, FENG W, WANG S. Self-supervised representation learning for skeleton-based group activity recognition[C]//Proceedings of the 30th ACM International Conference on Multimedia. New York, USA: ACM Press, 2022: 5990-5998.
35	BRUNA J, ZAREMBA W, SZLAM A, et al. Spectral networks and locally connected networks on graphs[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1312.6203.
36	KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1609.02907.
37	NIEPERT M, AHMED M, KUTZKOV K. Learning convolutional neural networks for graphs[C]// Proceedings of the 33rd International Conference on Machine Learning. New York, USA: ACM Press, 2016: 2014-2023.
38	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of Conference on Advances in Neural Information. Washington D. C., USA: IEEE Press, 2017: 76-85.
39	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washing D. C., USA: IEEE Press, 2016: 770-778.
40	LIU J, SHAHROUDY A, PEREZ M, et al. NTU RGB+D 120: a large-scale benchmark for 3D human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (10): 2684- 2701. doi: 10.1109/tpami.2019.2916873

[1]	李现国, 李滨. 基于Transformer和多尺度CNN的图像去模糊[J]. 计算机工程, 2023, 49(9): 226-233, 245.
[2]	苏晓东, 李世洲, 赵佳圆, 亮洪宇, 张玉荣, 徐红岩. 基于多级叠加和注意力机制的图像语义分割[J]. 计算机工程, 2023, 49(9): 265-271, 278.
[3]	韩璐, 霍纬纲, 张永会, 刘涛. 基于多尺度特征融合与双注意力机制的多元时间序列预测[J]. 计算机工程, 2023, 49(9): 99-108.
[4]	洪犇, 钱旭升, 申明磊, 胡冀苏, 耿辰, 戴亚康, 周志勇. 基于深度学习的CT-MR图像联合配准分割方法[J]. 计算机工程, 2023, 49(9): 234-245.
[5]	江雨燕, 陶承凤, 李平. 数据增强和自适应自步学习的深度子空间聚类算法[J]. 计算机工程, 2023, 49(8): 96-103, 110.
[6]	宋志娜, 李莎, 杨建明, 徐川. 基于特征与区域定位增强的遥感舰船目标检测[J]. 计算机工程, 2023, 49(8): 257-264.
[7]	张欣怡, 张飞, 郝斌, 高鹭, 任晓颖. 基于改进YOLOv5的口罩佩戴检测算法[J]. 计算机工程, 2023, 49(8): 265-274.
[8]	马建红, 龚天, 姚爽. 基于证据句与图卷积网络的文档级关系抽取[J]. 计算机工程, 2023, 49(8): 104-110.
[9]	马娜, 温廷新, 贾旭, 李晓会. 复杂光照条件下自适应的车脸重识别模型[J]. 计算机工程, 2023, 49(8): 275-282, 290.
[10]	汤卫芬, 高翠芳. 极值点自适应加权的动态时间规整算法[J]. 计算机工程, 2023, 49(7): 150-160.
[11]	梅晶, 戴龙宝, 童钊, 邓昕, 王嘉珂. 资源约束下基于Lyapunov优化的自适应卸载算法[J]. 计算机工程, 2023, 49(7): 34-46.
[12]	蔡倩倩, 史旭华. 自适应迁移的分解多目标多任务进化算法[J]. 计算机工程, 2023, 49(7): 55-64.
[13]	顾轶寅, 王鸿奎, 殷海兵. 基于上下文自适应阈值剪枝的快速依赖量化算法[J]. 计算机工程, 2023, 49(7): 143-149.
[14]	王款, 宣士斌, 何雪东, 李紫薇, 李嘉祥. 基于交叉注意力Transformer的人体姿态估计方法[J]. 计算机工程, 2023, 49(7): 223-231.
[15]	熊中敏, 曾旗, 卢鹏, 王振华, 郑宗生. 基于残差注意力多尺度关系网络的逻辑推理[J]. 计算机工程, 2023, 49(6): 227-233,241.

选择文件类型/文献管理软件名称

选择包含的内容