Skeleton Behavior Recognition Based on Extended Temporal and Spatiotemporal Feature Fusion Graph Convolutional Network

doi:10.19678/j.issn.1000-3428.0069044

Abstract

Abstract:

In recent years, significant progress has been made in the field of skeleton-based human behavior recognition using Graph Convolutional Networks (GCNs). However, most of the existing GCNs concatenate temporal and spatial convolutions in a straightforward manner, which leads to suboptimal spatiotemporal feature fusion. In addition, existing models face challenges in terms of efficiently extracting temporal features. To address these issues, this paper proposes an Extended Temporal and spatiotemporal Feature Fusion Graph Convolutional Network (ETFF-GCN). This network employs channel aggregation to fuse dynamic spatial topology and temporal features in a two-stage fusion process, followed by the application of attention mechanisms for further enhancement. In addition, multiple convolutional kernels of varying sizes are utilized to construct temporal graph convolutions for capturing multiscale and multigranular temporal characteristics. Furthermore, an effective compression excitation module is used for feature enhancement, which leads to improved feature representation capabilities. Experiments on three large datasets demonstrate that the proposed approach outperforms existing methods.

Key words: human skeleton behavior recognition, Graph Convolution Network(GCN), temporal and spatiotemporal feature fusion, attention mechanism, extended temporal

摘要：

在基于骨架的人体行为识别领域, 图卷积网络(GCN)在近年来取得了很大的进展, 但现有GCN大多将时间卷积和空间卷积简单串联, 导致时空特征融合效果不佳。另外, 现有模型还存在无法高效提取时间特征的问题。为此, 提出扩展时间和时空特征融合图卷积网络(ETFF-GCN)。该网络采用通道聚合的方法对动态空间拓扑和时序特征进行一次融合, 然后运用注意力机制进行二次融合, 进一步增强融合效果。在此基础上, 为了全面提取时序特征, 采用多个不同大小的卷积核构建时域图卷积, 以提取多尺度和多粒度的时间特征, 并引入有效压缩激励模块进行特征增强, 提升特征表达能力。在3个大型数据集上对所提出的方法进行评估, 实验结果表明, 该方法的性能优于现有的方法。

关键词: 人体骨架行为识别, 图卷积网络, 时空特征融合, 注意力机制, 扩展时间

XU Yonggang, SUN Qixuan, LI Fanjia, CHENG Jianwei, DAI Jiajun. Skeleton Behavior Recognition Based on Extended Temporal and Spatiotemporal Feature Fusion Graph Convolutional Network[J]. Computer Engineering, 2025, 51(4): 281-292.

徐永刚, 孙琦烜, 李凡甲, 程健维, 戴佳俊. 基于扩展时间和时空特征融合图卷积网络的骨架行为识别[J]. 计算机工程, 2025, 51(4): 281-292.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0069044

https://www.ecice06.com/EN/Y2025/V51/I4/281

Figures/Tables 20

Fig.1 Human body image and skeleton image

Fig.2 Overall architecture of ETFF-GCN

Fig.3 Structure of temporal and spatiotemporal feature fusion module

Fig.4 Structure of temporal and spatiotemporal aggregation graph convolution submodule

Fig.5 Structure of feature fusion submodule

Fig.6 Structure of expansion temporal convolution module

Fig.7 Performance and convergence speed of NTU RGB+D 60 dataset under X-Sub setting

Fig.8 Structure of four different models

Fig.9 Four different variant models

Fig.10 Feature visualization of different models

References 37

1	罗会兰, 曹立京. 基于多维动态拓扑学习图卷积的骨架动作识别. 电子学报, 2024, 52 (3): 991- 1001.
	LUO H L , CAO L J . Skeleton action recognition based on multidimensional dynamic topology learning graph volume product. Acta Electronica Sinica, 2024, 52 (3): 991- 1001.
2	建中华, 南静, 刘鑫, 等. 基于时空张量融合的人体骨架行为自适应识别方法. 仪器仪表学报, 2023, 44 (6): 74- 85.
	JIAN Z H , NAN J , LIU X , et al. Adaptive recognition method of human skeleton action with spatial-temporal tensor fusion. Chinese Journal of Scientific Instrument, 2023, 44 (6): 74- 85.
3	曹毅, 吴伟官, 李平, 等. 基于时空特征增强图卷积网络的骨架行为识别. 电子与信息学报, 2023, 45 (8): 3022- 3031.
	CAO Y , WU W G , LI P , et al. Skeleton action recognition based on spatio-temporal feature enhanced graph convolutional network. Journal of Electronics & Information Technology, 2023, 45 (8): 3022- 3031.
4	刘宽, 奚小冰, 周明东. 基于自适应多尺度图卷积网络的骨架动作识别. 计算机工程, 2023, 49 (10): 264- 271. doi: 10.19678/j.issn.1000-3428.0065882
	LIU K , XI X B , ZHOU M D . Skeleton action recognition based on adaptive multi-scale graph convolutional network. Computer Engineering, 2023, 49 (10): 264- 271. doi: 10.19678/j.issn.1000-3428.0065882
5	宋涛, 杨鑫, 马婧华, 等. 骨架自适应与关节增强图卷积网络的行为识别. 华中科技大学学报(自然科学版), 2022, 50 (3): 74-79, 93.
	SONG T , YANG X , MA J H , et al. Action recognition based on skeleton adaptation and joint enhancement graph convolutional network. Journal of Huazhong University of Science and Technology(Natural Science Edition), 2022, 50 (3): 74-79, 93.
6	赫磊, 邵展鹏, 张剑华, 等. 基于深度学习的行为识别算法综述. 计算机科学, 2020, 47 (S1): 139- 147.
	HE L , SHAO Z P , ZHANG J H , et al. Review of deep learning-based action recognition algorithms. Computer Science, 2020, 47 (S1): 139- 147.
7	YAN S J , XIONG Y J , LIN D H . Spatial temporal graph convolutional networks for skeleton-based action recognition. Artificial Intelligence, 2018, 32 (1): 7444- 7452.
8	THAKKAR K C, NARAYANAN P J. Part-based graph convolutional network for action recognition: supplementary material[EB/OL]. [2023-11-10]. https://arxiv.org/abs/1809.04983.
9	LI M S, CHEN S H, CHEN X, et al. Actional-structural graph convolutional networks for skeleton-based action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 3590-3598.
10	SHI L, ZHANG Y F, CHENG J, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 12018-12027.
11	CHEN Y X, ZHANG Z Q, YUAN C F, et al. Channel-wise topology refinement graph convolution for skeleton-based action recognition[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 13339-13348.
12	CHI H G, HA M H, CHI S, et al. InfoGCN: representation learning for human skeleton-based action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 20154-20164.
13	DAI Y M, GIESEKE F, OEHMCKE S, et al. Attentional feature fusion[C]//Proceedings of IEEE Winter Conference on Applications of Computer Vision. Washington D. C., USA: IEEE Press, 2021: 3559-3568.
14	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7132-7141.
15	SHAHROUDY A, LIU J, NG T T, et al. NTU RGB+D: a large scale dataset for 3D human activity analysis[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 1010-1019
16	LIU J , SHAHROUDY A , PEREZ M , et al. NTU RGB+D 120: a large-scale benchmark for 3D human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (10): 2684- 2701. doi: 10.1109/TPAMI.2019.2916873
17	WANG J, NIE X H, XIA Y, et al. Cross-view action modeling, learning, and recognition[C]//Proceedings of 2014 IEEE Conference on Computer Vision and Pattern Recognition. New York, USA: ACM Press, 2014: 2649-2656.
18	ZHANG P F, LAN C L, ZENG W J, et al. Semantics-guided neural networks for efficient skeleton-based human action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 1109-1118.
19	BARADEL F, WOLF C, MILLE J, et al. Glimpse clouds: human activity recognition from unstructured feature points[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 469-478.
20	CHENG K, ZHANG Y F, HE X Y, et al. Skeleton-based action recognition with shift graph convolutional network[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 180-189.
21	LI F J , ZHU A C , LI J J , et al. Frequency-driven channel attention-augmented full-scale temporal modeling network for skeleton-based action recognition. Knowledge-Based Systems, 2022, 256, 109854. doi: 10.1016/j.knosys.2022.109854
22	LIU Z Y, ZHANG H W, CHEN Z H, et al. Disentangling and unifying graph convolutions for skeleton-based action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 140-149.
23	SONG Y F , ZHANG Z , SHAN C , et al. Constructing stronger and faster baselines for skeleton-based action recognition. IEEE Transactions Pattern Anal Mach Intell, 2023, 45 (2): 1474- 1488. doi: 10.1109/TPAMI.2022.3157033
24	CHEN T L, ZHOU D S, WANG J, et al. Learning multi-granular spatio-temporal graph network for skeleton-based action recognition[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York, USA: ACM Press, 2021: 4334-4342.
25	QIU H L, HOU B, REN B, et al. Spatio-temporal tuples transformer for skeleton-based action recognition[EB/OL]. [2023-11-10]. https://arxiv.org/abs/2201.02849.
26	HU L Y, LIU S, FENG W. Spatial temporal graph attention network for skeleton-based action recognition[EB/OL]. [2023-11-10]. https://arxiv.org/abs/2208.08599.
27	WANG S Q, ZHANG Y J, ZHAO M H, et al. Skeleton-based action recognition via temporal-channel aggregation[EB/OL]. [2023-11-10]. https://arxiv.org/abs/2205.15936.
28	TRIVEDI N, SARVADEVABHATLA R K. PSUMNet: unified modality part streams are all you need forEfficient pose-based action recognition[C]//Proceedings of ECCV'22. Berlin, Germany: Springer, 2023: 211-227.
29	LIU J F , WANG X S , WANG C , et al. Temporal decoupling graph convolutional network for skeleton-based gesture recognition. IEEE Transactions on Multimedia, 2023, 26, 811- 823.
30	CHENG Q , CHENG J , REN Z L , et al. Multi-scale spatial-temporal convolutional neural network for skeleton-based action recognition. Pattern Analysis and Applications, 2023, 26 (3): 1303- 1315. doi: 10.1007/s10044-023-01156-w
31	ZHANG P , LAN C , XING J , et al. View adaptive neural networks for high performance skeleton-based human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41 (8): 1963- 1978. doi: 10.1109/TPAMI.2019.2896631
32	ZHU J G, ZOU W, XU L, et al. Action machine: rethinking action recognition in trimmed videos[EB/OL]. [2023-11-10]. https://arxiv.org/abs/1812.05770.
33	SI C Y, CHEN W T, WANG W, et al. An attention enhanced graph convolutional LSTM network for skeleton-based action recognition[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 1227-1236.
34	YU B X B , LIU Y , ZHANG X , et al. MMNet: a model-based multimodal network for human action recognition in RGB-D videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45 (3): 3522- 3538. doi: 10.1109/TPAMI.2022.3177813
35	CHENG K, ZHANG Y F, CAO C Q, et al. Decoupling GCN with DropGraph module for skeleton-based action recognition[C]//Proceedings of ECCV'20. Berlin, Germany: Springer, 2020: 536-553.
36	DAS S , DAI R , YANG D , et al. VPN++: rethinking video-pose embeddings for understanding activities of daily living. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 (12): 9703- 9717. doi: 10.1109/TPAMI.2021.3127885
37	XU K L, YE F F, ZHONG Q Y, et al. Topology-aware convolutional neural network for efficient skeleton-based action recognition[EB/OL]. [2023-11-10]. https://arxiv.org/abs/2112.04178.

[1]	XIE Qing, ZHANG Lingfeng, MA Yanchun, LIU Yongjian. Single Image Reflection Removal Model Based on Reflection Classifier and Gradient Restorer [J]. Computer Engineering, 2025, 51(4): 227-238.
[2]	YANG Ping, ZHANG Xi. Improved DeepLabv3+ Road Surface Crack Detection Method [J]. Computer Engineering, 2025, 51(4): 261-270.
[3]	GENG Xia, WANG Yao. Cloth-Changing Person Re-Identification Method Based on CLIP Enhanced Fine-Grained Features [J]. Computer Engineering, 2025, 51(4): 293-302.
[4]	LIU Yunxiang, LIANG Zhichao. A Highly Efficient Traffic Prediction Model for Continuous Time-series Graph Attention Networks [J]. Computer Engineering, 2025, 51(4): 350-359.
[5]	DONG Hongliang, NIU Yan, SUN Yang, LI Jun. Speech Emotion Recognition Based on Memory Capsules and Attention [J]. Computer Engineering, 2025, 51(4): 169-177.
[6]	SUN Ziwen, QIAN Lizhi, YUAN Guanglin, YANG Chuandong, LING Chong. Transformer Object Tracking Method Based on Real-Time Dynamic Template Update [J]. Computer Engineering, 2025, 51(4): 158-168.
[7]	DU Chenyang, ZHANG Xueying, HUANG Lixia, LI Juan. Multi-Feature Speech Emotion Recognition Based on Improved Efficient Channel Attention Mechanism [J]. Computer Engineering, 2025, 51(4): 97-106.
[8]	LUAN Fangjun, GONG Qi, YUAN Shuai. Crowd Counting Network Based on Attention Mechanism and Multiscale Fusion [J]. Computer Engineering, 2025, 51(3): 352-361.
[9]	SUN Ting, YANG Jie, LI Jiaxuan, WANG Yaozong. Optimization of YOLOv7 Road Sign Detection Algorithm for Low-Light Traffic Scenes [J]. Computer Engineering, 2025, 51(3): 342-351.
[10]	HU Shulin, ZHANG Huajun, DENG Xiaotao, WANG Zhenghua. Similarity Calculation for Chinese Text Based on Dependency Graph Convolution [J]. Computer Engineering, 2025, 51(3): 76-85.
[11]	LU Peng, ZHONG Chuang. Improved CycleGAN Algorithm for Semi-Supervised Building Extraction [J]. Computer Engineering, 2025, 51(3): 241-251.
[12]	WANG Xinliang, WANG Luying. Safety Helmet Detection Algorithm with Feature Enhancement in Low Light Blasting Scenes [J]. Computer Engineering, 2025, 51(3): 252-260.
[13]	ZHANG Xingpeng, HE Dong, YANG Mo, YE Hangbin. Nucleus Segmentation Based on Multiscale Attention and Data Augmentation [J]. Computer Engineering, 2025, 51(2): 387-396.
[14]	XU Ming, QU Taipeng, JIANG Yanji. Improved YOLOv7 Traffic Sign Detection Algorithm in Complex Scenarios [J]. Computer Engineering, 2025, 51(2): 335-343.
[15]	ZHANG Yuan, LÜ Defang, MENG Jianjun, QI Wenzhe. Defect Detection of Rail Fasteners Based on Double Attention and GSSN Lightweight [J]. Computer Engineering, 2025, 51(2): 289-299.

Please choose a citation manager

Content to export