基于多流语义图卷积网络的人体行为识别

doi:10.19678/j.issn.1000-3428.0067977

摘要/Abstract

摘要：

与基于图像的行为识别方法相比, 利用人体骨架信息进行识别能有效克服复杂背景、光照变化以及外貌变化等因素影响。但是, 目前主流的基于人体骨架的行为识别方法存在参数量过大、运算速度慢等问题。对此, 提出一种多流轻量级语义图卷积的行为识别方法。设计多流语义引导的图卷积网络(MS-SGN), 将骨架信息分别表达为骨长流、关节流和细粒度流3种数据流形式, 再对嵌入语义信息的数据流通过自适应图卷积提取空间特征, 并采用不同内核和膨胀率的多尺度时域卷积提取时域特征, 最后对各流分类结果进行加权融合。实验结果表明, 该方法在NTU60 RGB+D数据集上的识别精度分别为90.0%(X-Sub)和95.83%(X-View), 在NTU120 RGB+D数据集上的识别精度分别为83.4%(X-Sub)和84.0%(X-View), 优于SGN、Logsin-RNN等主流方法, 且网络框架更为轻量化。

关键词: 行为识别, 人体骨架, 特征融合, 图卷积网络, 多尺度

Abstract:

Compared to image-based behavior recognition methods, utilizing human skeleton information for recognition effectively overcomes the influence of complex backgrounds, lighting changes, and appearance variations. However, most mainstream human skeleton based methods encounter issues such as large parameter quantity and slow computational speed. To address these issues, this paper proposes a Multi-Stream lightweight Semantic Graph Convolutional Network(MS-SGN) for behavior recognition. The skeleton information is expressed as three data streams: bone length flow, joint flow, and fine-grained flow. Spatial features are extracted from the data stream embedded with semantic information through adaptive graph convolution. Time-domain features are extracted using multi-scale time-domain convolution with different kernels and expansion rates. Finally, the classification results of each stream are weighted and fused. The proposed method achieves an accuracy of 90.0%(X-Sub) and 95.83%(X-View) on large-scale dataset NTU60 RGB+D, and 83.4%(X-Sub) and 84.0%(X-View) on the dataset NTU120 RGB+D, respectively. Comparative experiments demonstrate that the proposed method offers better recognition accuracy than several main methods such as SGN and Logsin-RNN, while maintaining a lightweight network framework.

Key words: behavior recognition, human skeleton, feature fusion, Graph Convolutional Network(GCN), multi-scale

刘锁兰, 王炎, 王洪元, 朱生升. 基于多流语义图卷积网络的人体行为识别[J]. 计算机工程, 2024, 50(8): 64-74.

Suolan LIU, Yan WANG, Hongyuan WANG, Shengsheng ZHU. Human Behavior Recognition Based on Multi-Stream Semantic Graph Convolutional Network[J]. Computer Engineering, 2024, 50(8): 64-74.

https://www.ecice06.com/CN/Y2024/V50/I8/64

图/表 15

图1 拓扑矩阵图

Fig.1 Topological matrix diagrams

图2 MS-SGN框架

Fig.2 Framework of MS-SGN

图3 多流数据

Fig.3 Multi-stream data

图4 语义图卷积网络基本模块

Fig.4 Basic modules of semantic graph convolutional network

图5 NTU RGB+D数据集数据示例

Fig.5 Examples of data in NTU RGB+D dataset

图6 训练精度曲线

Fig.6 Training accuracy curve

图7 骨长流数据与细粒度流数据可视化

Fig.7 Visualization of bone long stream data and fine-grained stream data

图8 3种数据流在典型动作上的分类得分

Fig.8 Classification scores of three stream data on typical actions

图9 流数据训练精度离散图

Fig.9 Discrete graphs of stream data training accuracy

参考文献 34

1	SHI L, ZHOU Y M, WANG J, et al. Compact global association based adaptive routing framework for personnel behavior understanding. Future Generation Computer Systems, 2023, 141, 514- 525. doi: 10.1016/j.future.2022.12.002
2	YAN S, XIONG X H, ARNAB A, et al. Multiview transformers for video recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 3333-3343.
3	鹿天然, 于凤芹, 陈莹. 一种基于线性序列差异分析降维的人体行为识别方法. 计算机工程, 2019, 45(3): 237-241, 249. URL
	LU T R, YU F Q, CHEN Y. A human action recognition method based on LSDA dimension reduction. Computer Engineering, 2019, 45(3): 237-241, 249. URL
4	YU J H, WANG Z R, VASUDEVAN V, et al. CoCa: contrastive captioners are image-text foundation models[EB/OL]. [2023-05-10]. https://arxiv.org/abs/2205.01917.
5	王恩德, 刘巧英, 李勇. 基于LLC与GIST特征的静态人体行为分类. 计算机工程, 2018, 44(8): 268-272, 278. URL
	WANG E D, LIU Q Y, LI Y. Static human behavior classification based on LLC and GIST features. Computer Engineering, 2018, 44(8): 268-272, 278. URL
6	REN Z L, ZHANG Q S, CHENG J, et al. Segment spatial-temporal representation and cooperative learning of convolution neural networks for multimodal-based action recognition. Neurocomputing, 2021, 433, 142- 153. doi: 10.1016/j.neucom.2020.12.020
7	宋真东, 杨国超, 马玉鹏, 等. 基于注意力机制的多模态人体行为识别算法. 计算机测量与控制, 2022, 30(2): 276- 283. URL
	SONG Z D, YANG G C, MA Y P, et al. Multi-modal human behavior recognition algorithm based on attention mechanism. Computer Measurement & Control, 2022, 30(2): 276- 283. URL
8	WANG L M, XIONG Y J, WANG Z, et al. Temporal segment networks: towards good practices for deep action recognition[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 20-36.
9	GU Y, SHENG W H, OU Y S, et al. Human action recognition with contextual constraints using a RGB-D sensor[C]//Proceedings of the IEEE International Conference on Robotics and Biomimetics. Washington D. C., USA: IEEE Press, 2013: 674-679.
10	LIU J, SHAHROUDY A, XU D, et al. Spatio-temporal LSTM with trust gates for 3D human action recognition[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 816-833.
11	YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition[EB/OL]. [2023-05-10]. https://arxiv.org/abs/1801.07455.
12	LI M S, CHEN S H, CHEN X, et al. Actional-structural graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 3595-3603.
13	CHEN Y X, ZHANG Z Q, YUAN C F, et al. Channel-wise topology refinement graph convolution for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 13359-13368.
14	ZHANG P F, LAN C L, ZENG W J, et al. Semantics-guided neural networks for efficient skeleton-based human action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 1112-1121.
15	ZHANG J X, YE G X, TU Z G, et al. A spatial attentive and temporal dilated(SATD) GCN for skeleton-based action recognition. CAAI Transactions on Intelligence Technology, 2022, 7(1): 46- 55. doi: 10.1049/cit2.12012
16	SHI L, ZHANG Y F, CHENG J, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 12026-12035.
17	CHENG K, ZHANG Y F, HE X Y, et al. Skeleton-based action recognition with shift graph convolutional network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 183-192.
18	XIE J, MIAO Q G, LIU R Y, et al. Attention adjacency matrix based graph convolutional networks for skeleton-based action recognition. Neurocomputing, 2021, 440, 230- 239. doi: 10.1016/j.neucom.2021.02.001
19	CHI H G, HA M H, CHI S, et al. InfoGCN: representation learning for human skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 20186-20196.
20	YANG S Y, LIU J, LU S J, et al. Skeleton cloud colorization for unsupervised 3D action representation learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 13423-13433.
21	HEDEGAARD L, HEIDARI N, IOSIFIDIS A. Continual spatio-temporal graph convolutional networks. Pattern Recognition, 2023, 140, 109528. doi: 10.1016/j.patcog.2023.109528
22	KIM T S, REITER A. Interpretable 3D human action analysis with temporal convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 1623-1631.
23	DEFFERRARD M, BRESSON X, VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2016: 3844-3852.
24	van den OORD A, KALCHBRENNER N, KAVUKCUOGLU K. Pixel recurrent neural networks[C]//Proceedings of the 33rd International Conference on Machine Learning. New York, USA: ACM Press, 2016: 2611-2620.
25	WANG H S, WANG L. Modeling temporal dynamics and spatial configurations of actions using two-stream recurrent neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 499-508.
26	TU Z G, ZHANG J X, LI H Y, et al. Joint-bone fusion graph convolutional network for semi-supervised skeleton action recognition. IEEE Transactions on Multimedia, 2023, 25, 1819- 1831. doi: 10.1109/TMM.2022.3168137
27	ZHANG P F, LAN C L, XING J L, et al. View adaptive neural networks for high performance skeleton-based human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(8): 1963- 1978. doi: 10.1109/TPAMI.2019.2896631
28	SHI L, ZHANG Y F, CHENG J, et al. AdaSGN: adapting joint number and model size for efficient skeleton-based action recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 13413-13422.
29	PLIZZARI C, CANNICI M, MATTEUCCI M. Spatial temporal transformer network for skeleton-based action recognition[C]//Proceedings of International Conference on Pattern Recognition. New York, USA: ACM Press, 2021: 694-701.
30	CAETANO C, BREMOND F, SCHWARTZ W R. Skeleton image representation for 3D action recognition based on tree structure and reference joints[C]//Proceedings of the 32nd SIBGRAPI Conference on Graphics, Patterns and Images. Washington D. C., USA: IEEE Press, 2019: 16-23.
31	LIAO S, LYONS T, YANG W, et al. Learning stochastic differential equations using RNN with log signature features[EB/OL]. [2023-05-10]. https://arxiv.org/pdf/1908.08286v2.
32	MEMMESHEIMER R, THEISEN N, PAULUS D. Gimme Signals: discriminative signal encoding for multimodal activity recognition[C]//Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. Washington D. C., USA: IEEE Press, 2020: 10394-10401.
33	PENG W, SHI J G, XIA Z Q, et al. Mix Dimension in Poincaré geometry for 3D skeleton-based action recognition[C]//Proceedings of the 28th ACM International Conference on Multimedia. New York, USA: ACM Press, 2020: 1432-1440.
34	SONG Y F, ZHANG Z, SHAN C F, et al. Richly activated graph convolutional network for robust skeleton-based action recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(5): 1915- 1925. doi: 10.1109/TCSVT.2020.3015051

[1]	汤志康, 武毓琦, 李春英, 汤庸. 基于知识图谱卷积网络的学习资源推荐[J]. 计算机工程, 2024, 50(9): 153-160.
[2]	李俊仪, 李向阳, 龙朝勋, 李海燕, 李红松, 余鹏飞. 基于多级区域选择与跨层特征融合的野生菌分类[J]. 计算机工程, 2024, 50(9): 179-188.
[3]	朱凯, 李理, 张彤, 江晟, 别一鸣. 基于Transformer的多阶段运动模糊图像修复网络[J]. 计算机工程, 2024, 50(9): 276-285.
[4]	高煜宝, 文志诚. 基于注意力机制的双路解码器图像去噪方法[J]. 计算机工程, 2024, 50(9): 324-332.
[5]	潘恩元, 钟原, 李平. 联邦异质性数据下半监督颈椎MRI分割模型[J]. 计算机工程, 2024, 50(9): 367-376.
[6]	张华青, 夏张涛, 陆晓庆, 童基均. 基于字形特征的血管外科命名实体识别[J]. 计算机工程, 2024, 50(8): 13-21.
[7]	李华昱, 张智康, 闫阳, 岳阳. 基于知识图谱增强的领域多模态实体识别[J]. 计算机工程, 2024, 50(8): 31-39.
[8]	赵婉秋, 张俊虎, 李海涛. 用于建筑物分割的平行结构特征融合网络[J]. 计算机工程, 2024, 50(8): 239-248.
[9]	赵宏, 王枭. 基于Swin-Transformer的黑色素瘤图像病灶分割研究[J]. 计算机工程, 2024, 50(8): 249-258.
[10]	王富平, 刘鸿玮, 张锲石, 段冠庄. 基于深度特征抑制的遮挡人脸识别网络[J]. 计算机工程, 2024, 50(8): 259-269.
[11]	闵莉, 董冰洁, 安冬. 基于多注意力机制与跨特征融合的语义分割算法[J]. 计算机工程, 2024, 50(8): 282-289.
[12]	陈宇航, 杨勇, 先木斯亚·买买提明, 帕力旦·吐尔逊, 樊小超, 任鸽, 刁宇峰. 基于主题感知和语义增强的作文自动评分方法[J]. 计算机工程, 2024, 50(8): 363-371.
[13]	谭巨全, 王然. 特征融合下田径录像3D人体动作DTW捕捉算法[J]. 计算机工程, 2024, 50(7): 71-78.
[14]	张溢文, 蔡满春, 陈咏豪, 朱懿, 姚利峰. 融合空间特征的多尺度深度伪造检测方法[J]. 计算机工程, 2024, 50(7): 240-250.
[15]	李云航, 潘晴, 田妮莉. 结构相似度优化的混合多尺度医学图像融合[J]. 计算机工程, 2024, 50(7): 264-270.

选择文件类型/文献管理软件名称

选择包含的内容