基于图卷积的局部特征细化动作识别方法

doi:10.19678/j.issn.1000-3428.0068663

摘要/Abstract

摘要：

动作识别是计算机视觉领域一个重要研究方向。目前, 主流方法在局部动作特征上的关注度不足。部分动作识别方法为关注局部动作特征, 将预定义的人体骨架拆分成左右手、左右腿等多个部分。但是, 这些部分包含的骨架关键点较少, 使得动作特征较相似, 导致识别准确率降低。此外, 已有基于局部动作特征的动作识别方法未充分考虑全局姿态特征, 模型识别准确率不稳定。为此, 提出基于图卷积的局部特征细化动作识别方法。将预定义人体骨骼拓扑图划分为身体、上下肢, 加强模型关注局部动作特征的能力。同时, 设计局部特征细化器, 采用对比学习策略扩大不同种类动作的局部动作特征差异, 缩小同类动作之间的差异, 解决因划分策略造成动作特征相似的问题。在此基础上, 将上下肢与身体的分类结果相结合, 充分利用全局姿态特征, 提高模型的稳定性。实验结果表明, 该方法在NTU RGB+D 60 2个基准数据集X-Sub、X-View的识别准确率分别为93.0%和98.8%, 在NTU RGB+D 120 2个基准数据集X-Sub、X-Set的识别准确率分别为88.8%和90.1%, 能够有效提高动作识别的准确率。

关键词: 动作识别, 对比学习, 骨骼关键点, 预定义骨骼拓扑, 局部特征细化

Abstract:

Although action recognition is an important research area in computer vision, current mainstream methods lack a sufficient emphasis on local features. Some action recognition approaches focus on local action features by dividing the predefined human skeleton into various parts, such as the left and right hands, left and right legs. However, these parts contain fewer skeleton keypoints, resulting in similar action features and a lower recognition efficiency. Moreover, existing methods based on local action features often neglect global posture characteristics, leading to unstable model recognition accuracy. To address these issues, this study proposes a method for refining local features in action recognition based on graph convolution. The proposed method divides the predefined human skeleton topology into body and upper/lower limbs, enhancing the model's capability to focus on local action features. Simultaneously, a local feature refiner uses contrastive learning strategies to expand the differences in the local action features of different types of actions, reduce the differences between similar actions, and solve the problem of similar action features caused by partitioning strategies. Accordingly, the classification results of the upper and lower limbs are combined with those of the body, fully utilizing the global pose features to improve model stability. Experimental results show that the recognition accuracies achieved by this method on two NTU RGB+D 60 benchmark datasets X-Sub and X-View are 93.0% and 98.8%, respectively. Furthermore, the recognition accuracies of X-Sub and X-Set on the NTU RGB+D 120 benchmark datasets are 88.8% and 90.1%, respectively, representing effective improvements in the accuracy of action recognition.

Key words: action recognition, contrastive learning, skeleton keypoints, predefined skeleton topology, local feature refinement

贺子泽, 战荫伟. 基于图卷积的局部特征细化动作识别方法[J]. 计算机工程, 2024, 50(11): 276-283.

HE Zize, ZHAN Yinwei. Local Feature Refinement Action Recognition Method Based on Graph Convolution[J]. Computer Engineering, 2024, 50(11): 276-283.

https://www.ecice06.com/CN/Y2024/V50/I11/276

图/表 10

图1 局部生成器示意图

Fig.1 Schematic diagram of local generator

图2 时空特征提取模块

Fig.2 Spatio-temporal feature extraction module

图3 动作识别网络架构

Fig.3 Architecture of action recongnition network

图4 局部细化模块

Fig.4 Local refinement module

图5 Pa-ResGCN与本文方法的浅层特征可视化图

Fig.5 Visualization of latent feature between Pa-ResGCN and the method proposed in this paper

参考文献 23

1	DUAN H D, ZHAO Y, CHEN K, et al. Revisiting skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 2969-2978.
2	YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition[EB/OL]. [2023-09-18]. https://arxiv.org/pdf/1801.07455.
3	LIU Z Y, ZHANG H W, CHEN Z H, et al. Disentangling and unifying graph convolutions for skeleton-based action recognition [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 143-152.
4	CHEN Y X, ZHANG Z Q, YUAN C F, et al. Channel-wise topology refinement graph convolution for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 13359-13368.
5	SHI L, ZHANG Y F, CHENG J, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition[EB/OL]. [2023-09-18]. https://arxiv.org/pdf/1805.07694.
6	DU Y, WANG W, WANG L. Hierarchical recurrent neural network for skeleton based action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2015: 1110-1118.
7	THAKKAR K, NARAYANAN P J. Part-based graph convolutional network for action recognition[EB/OL]. [2023-09-18]. https://arxiv.org/pdf/1809.04983.
8	SONG Y F, ZHANG Z, SHAN C F, et al. Stronger, faster and more explainable: a graph convolutional baseline for skeleton-based action recognition[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York, USA: ACM Press, 2020: 1625-1633.
9	程思雨, 陈莹. 伪标签细化引导的相机感知无监督行人重识别方法. 光电工程, 2023, 50 (12): 230239. URL
	CHENG S Y, CHEN Y. Camera-aware unsupervised person re-identification method guided by pseudo-label refinement. Opto-Electronic Engineering, 2023, 50 (12): 230239. URL
10	CHEN X L, FAN H Q, GIRSHICK R, et al. Improved baselines with momentum contrastive learning[EB/OL]. [2023-09-18]. https://arxiv.org/abs/2003.04297.
11	CHEN X, XIE S, HE K M. An empirical study of training self-supervised vision transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 9640-9649.
12	HE K M, FAN H Q, WU Y X, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 9729-9738.
13	WANG H, CHU H, SIMING F U, et al. Renovate yourself: calibrating feature representation of misclassified pixels for semantic segmentation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S. l.]: AAAI Press, 2022: 2450-2458.
14	SHAHROUDY A, LIU J, NG T T, et al. NTU RGB+D: a large scale dataset for 3D human activity analysis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 1010-1019.
15	LIU J, SHAHROUDY A, PEREZ M, et al. NTU RGB+D 120: a large-scale benchmark for 3D human activity understanding. IEEE Transactions on Pattern Analysis And Machine Intelligence, 2019, 42 (10): 2684- 2701.
16	WANG J, NIE X H, XIA Y, et al. Cross-view action modeling, learning and recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2014: 2649-2656.
17	CHENG K, ZHANG Y F, CAO C Q, et al. Decoupling GCN with DropGraph module for skeleton-based action recognition [C]//Proceedings of the 16th European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 536-553.
18	PLIZZARI C, CANNICI M, MATTEUCCI M. Skeleton-based action recognition via spatial and temporal transformer networks. Computer Vision and Image Understanding, 2021, 208, 103219. doi: 10.1016/J.CVIU.2021.103219
19	LEE H, PARK U, KIM I-J, et al. Rank-GCN for robust action recognition. IEEE Access, 2022, 10, 91739- 91749. doi: 10.1109/ACCESS.2022.3202164
20	LI C, HUANG Q, MAO Y. DD-GCN: directed diffusion graph convolutional network for skeleton-based human action recognition[C]//Proceedings of IEEE International Conference on Multimedia and Expo. Washington D. C., USA: IEEE Press, 2023: 786-791.
21	CHENG Q, CHENG J, REN Z L, et al. Multi-scale spatial-temporal convolutional neural network for skeleton-based action recognition. Pattern Analysis and Applications, 2023, 26, 1- 13. doi: 10.1007/s10044-022-01091-2
22	SONG Y F, ZHANG Z, SHAN C F, et al. Constructing stronger and faster baselines for skeleton-based action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45 (2): 1474- 1488. doi: 10.48550/arXiv.2106.15125
23	CHENG K, ZHANG Y F, HE X Y, et al. Skeleton-based action recognition with shift graph convolutional network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 183-192.

[1]	高爽, 史轶伦, 徐巧枝, 于磊. 基于对比学习的非对称编解码结构的心脏MRI分割研究[J]. 计算机工程, 2024, 50(8): 290-300.
[2]	钱来, 赵卫伟. 基于对比学习和注意力机制的文本分类方法[J]. 计算机工程, 2024, 50(7): 104-111.
[3]	刘娟, 段友祥, 陆誉翕, 张鲁. 引入知识增强和对比学习的知识图谱补全[J]. 计算机工程, 2024, 50(7): 112-122.
[4]	武星, 殷浩宇, 姚骏峰, 李卫民, 钱权. 面向视频数据的多模态情感分析[J]. 计算机工程, 2024, 50(6): 218-227.
[5]	张宝鑫, 杨丹, 聂铁铮, 寇月. 基于自监督的多视角图协同过滤推荐方法[J]. 计算机工程, 2024, 50(5): 100-110.
[6]	李田芳, 普园媛, 赵征鹏, 徐丹, 钱文华. 基于CLIP和双空间自适应归一化的图像翻译[J]. 计算机工程, 2024, 50(5): 229-240.
[7]	张洪程, 李林育, 杨莉, 伞晨峻, 尹春林, 颜冰, 于虹, 张璇. 基于对比学习与语言模型增强嵌入的知识图谱补全[J]. 计算机工程, 2024, 50(4): 168-176.
[8]	李政学, 李枝名, 彭德中, 陈杰. 基于特征对比学习和图卷积的社交网络用户分类[J]. 计算机工程, 2024, 50(4): 258-266.
[9]	吴冠荣, 李元祥, 王艺霖, 陆雨寒, 陈秀华. 基于对比学习的小样本金属表面损伤分类[J]. 计算机工程, 2024, 50(3): 36-43.
[10]	安国成, 江波, 王晓龙, 戴军. 基于拓展图文对比学习的多模态语义对齐[J]. 计算机工程, 2024, 50(11): 152-162.
[11]	吴永庆, 王钰涵, 朱月. 基于用户多类型反馈行为序列的点击率预估模型[J]. 计算机工程, 2024, 50(10): 405-417.
[12]	顾嘉静, 杨丹, 聂铁铮, 寇月. 基于多视图融合跨层对比学习的推荐算法[J]. 计算机工程, 2024, 50(1): 120-128.
[13]	曹发鑫, 孙媛媛, 王治政, 潘丁豪, 林鸿飞. 面向借贷案件的相似案例匹配模型[J]. 计算机工程, 2024, 50(1): 306-312.
[14]	李晓腾, 张盼盼, 勾智楠, 高凯. 基于多任务学习的多模态命名实体识别方法[J]. 计算机工程, 2023, 49(4): 114-119.
[15]	刘宽, 奚小冰, 周明东. 基于自适应多尺度图卷积网络的骨架动作识别[J]. 计算机工程, 2023, 49(10): 264-271.

选择文件类型/文献管理软件名称

选择包含的内容