Residual Behavior Recognition Model Based on Spatio-Temporal Shuffle Attention Mechanism

doi:10.19678/j.issn.1000-3428.0068936

Abstract

Abstract:

This paper presents a residual behavior recognition model based on Spatio-temporal Shuffle Attention(SAT) mechanism, to improve the effectiveness of 3D convolution extraction of spatio-temporal features in deep learning models. The SAT mechanism is a lightweight multidimensional hybrid attention mechanism composed of submoudule that combines channels and temporal attention and spatial attention submodule, which adds the dimension of time combination to obtain time and channel information in channel attention. The spatial attention submodule compresses redundant time information, improves the attention to spatial features, carries out channel scrambling and reorganization on extracted features, improves the data representation ability of the model, and reduces the parameter count. In this model, a Resnext residual network is used to extract spatio-temporal features, the spatio-temporal permutation attention mechanism module is embedded into the residual module, and the attention module is used to independently learn the weight parameters of different feature maps. The extracted features are weighted in the channel, time, and space domains to enhance the network's ability to express human behavior, and Focal Loss, which is an improved cross-entropy function, is used as the loss function to solve the uneven sample distribution problem in datasets. Experimental results show that the model achieves a recognition accuracy of 96.3% and 71.6% on the UCF101 and HMDB51 datasets, respectively, which is a significant improvement over other models.

Key words: deep learning, behavior recognition, Spatio-temporal Shuffle Attention (SAT), residual network, cross-entropy function

摘要：

为提升深度学习模型中三维卷积提取时空特征的有效性, 提出一种基于时空置换注意力(SAT)机制的残差行为识别模型。SAT机制是由通道结合时间和空间注意力子模块组成的轻量化的多维度混合注意力机制, 其在通道注意力中增加了结合时间的维度, 获取时间与通道信息; 在空间注意力中压缩冗余的时间信息, 提升对空间特征的关注度, 对提取的特征进行通道置乱及通道重组, 提升模型对数据的表征能力并减少参数量。该模型使用Resnext残差网络提取时空特征, 在残差模块中嵌入SAT模块, 利用注意力模块自主学习不同特征图的权重参数, 针对性地对提取的特征在通道、时间、空间域加权, 增强网络对人体行为的表达能力, 使用改进的交叉熵函数Focal Loss作为损失函数, 解决数据集中可能存在的样本分布不均衡的问题。实验结果表明, 该模型在UCF101以及HMDB51数据集上的识别准确率分别达到了96.3%以及71.6%, 相较于其他对比模型均有显著提升。

关键词: 深度学习, 行为识别, 时空置换注意力, 残差网络, 交叉熵函数

JIANG Jieping, WANG Mingwen. Residual Behavior Recognition Model Based on Spatio-Temporal Shuffle Attention Mechanism[J]. Computer Engineering, 2025, 51(4): 119-128.

蒋杰平, 王明文. 基于时空置换注意力机制的残差行为识别模型[J]. 计算机工程, 2025, 51(4): 119-128.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0068936

https://www.ecice06.com/EN/Y2025/V51/I4/119

Figures/Tables 16

Fig.1 Comparison of network structure

Fig.2 Resnext residual module structure

Fig.3 Structure of shuffle attention mechanism

Fig.4 Submodule that combines channels and temporal attention

Fig.5 Spatial attention submodule

Fig.6 Structure of SAT-Bottlenek

Fig.7 Structure of SAT-Resnext

Fig.8 Experiment procedure

Fig.9 UCF101 training process

Fig.10 HMDB51 training process

Fig.11 Overall confusion matrix diagram

Fig.12 Partial confusion matrix diagram

References 25

1	邓淼磊, 高振东, 李磊, 等. 基于深度学习的人体行为识别综述. 计算机工程与应用, 2022, 58 (13): 14- 26.
	DENG M L , GAO Z D , LI L , et al. Overview of human behavior recognition based on deep learning. Computer Engineering and Applications, 2022, 58 (13): 14- 26.
2	WANG H, SCHMID C. Action recognition with improved trajectories[C]//Proceedings of the IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2013: 3551-3558.
3	KARPATHY A, TODERICI G, SHETTY S, et al. Large-scale video classification with convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2014: 1725-1732.
4	SIMONYAN K, ZISSERMAN A, SIMONYAN K, et al. Two-stream convolutional networks for action recognition in videos[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2014: 568-576.
5	XIE Z , ZHOU Y , WU K W , et al. Behavior recognition based on spatiotemporal attention LSTM. Journal of Computer Science, 2021, 44 (2): 261- 274.
6	TRAN D, BOURDEV L, FERGUS R, et al. Learning spatiotemporal features with 3D convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2015: 4489-4497.
7	QIU Z F, YAO T, MEI T. Learning spatio-temporal representation with pseudo-3D residual networks[C]//Proceedings of the IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 5534-5542.
8	DIBA A L, FAYYAZ M, SHARMA V, et al. Temporal 3D ConvNets: new architecture and transfer learning for video classification[EB/OL]. [2023-09-18]. https://arxiv.org/abs/1711.08200v1.
9	HARA K, KATAOKA H, SATOH Y. Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet?[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 6546-6555.
10	CHAUDHARI S , MITHAL V , POLATKAN G , et al. An attentive survey of attention models. ACM Transactions on Intelligent Systems and Technology, 2021, 12 (5): 1- 32.
11	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7132-7141.
12	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 3-19.
13	CAO Y, XU J R, LIN S, et al. GCNet: non-local networks meet squeeze-excitation networks and beyond[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 1971-1980.
14	WANG Q, WU B, ZHU P, et al. ECA-Net: efficient channel attention for deep convolutional neural networks[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 11531-11539.
15	LI X, HU X L, YANG J. Spatial group-wise enhance: improving semantic feature learning in convolutional networks[EB/OL]. [2023-09-18]. https://arxiv.org/abs/1905.09646v2.
16	ZHOU Y D, LI B P, WANG Z H, et al. Integrating temporal and spatial attention for video action recognition[EB/OL]. [2023-09-18]. https://pdfs.semanticscholar.org/dd07/53f2192c52672a9b421c61abf58a47f8de47.pdf.
17	CARVALHO S R, BERTAGNOLLI N M, FOLKMAN T, et al. A temporal bottleneck attention architecture for video action recognition: WO2021US59372[P]. 2022-05-19.
18	LI C H , ZHANG J , YAO J C . Streamer action recognition in live video with spatial-temporal attention and deep dictionary learning. Neurocomputing, 2021, 453, 383- 392. doi: 10.1016/j.neucom.2020.07.148
19	GONG J , LUO C , LUO Q . Action recognition model based on attention mechanism and residual network. Electronic Measurement Technology, 2021, 44 (14): 111- 116.
20	XIE S N, GIRSHICK R, DOLLÁR P, et al. Aggregated residual transformations for deep neural networks[EB/OL]. [2023-09-18]. https://arxiv.org/abs/1611.05431.
21	ZHANG Q L, YANG Y B. SA-Net: shuffle attention for deep convolutional neural networks[EB/OL]. [2023-09-18]. https://arxiv.org/abs/2102.00240.
22	ZHANG X Y, ZHOU X Y, LIN M X, et al. ShuffleNet: an extremely efficient convolutional neural network for mobile devices[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 6848-6856.
23	LIN T Y , GOYAL P , GIRSHICK R , et al. Focal loss for dense object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42 (2): 318- 327. doi: 10.1109/TPAMI.2018.2858826
24	ZHOU B , LI J F . Human behavior recognition combined with object detection. Journal of Automation, 2020, 46 (9): 1961- 1970.
25	WANG L M , XIONG Y J , WANG Z , et al. Temporal segment networks for action recognition in videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41 (11): 2740- 2755.

[1]	XU Yonggang, SUN Qixuan, LI Fanjia, CHENG Jianwei, DAI Jiajun. Skeleton Behavior Recognition Based on Extended Temporal and Spatiotemporal Feature Fusion Graph Convolutional Network [J]. Computer Engineering, 2025, 51(4): 281-292.
[2]	DU Chenyang, ZHANG Xueying, HUANG Lixia, LI Juan. Multi-Feature Speech Emotion Recognition Based on Improved Efficient Channel Attention Mechanism [J]. Computer Engineering, 2025, 51(4): 97-106.
[3]	DAI Kangjia, XU Huiying, ZHU Xinzhong, LI Xiyu, HUANG Xiao, CHEN Guoqiang, ZHANG Zhixiong. YGL-SLAM: Point and Line Based Semantic SLAM System for Dynamic Scenes [J]. Computer Engineering, 2025, 51(3): 95-104.
[4]	HAN Peng, HUANG Yunzhi, REN Caiyue, CHENG Jingyi, XU Jun. Assessment of Neoadjuvant Chemotherapy Efficacy in Breast Cancer Using Dual-Branch Network with PET Imaging [J]. Computer Engineering, 2025, 51(3): 293-299.
[5]	HU Chaoju, GUO Fengyi. MODF Port State Detection Algorithm Based on Improved YOLOv7 [J]. Computer Engineering, 2025, 51(2): 78-85.
[6]	SUN Haomiao, LI Zongmin, XIAO Qian, SUN Wenjie, ZHANG Wenxin. AI-Curling: An On-Site Curling Analysis and Decision-Making Method [J]. Computer Engineering, 2025, 51(2): 102-110.
[7]	SUN Yikang, GAO Jianhua. Dead Code Detection Method Based on Convolutional Neural Network and Long Short-Term Memory [J]. Computer Engineering, 2025, 51(2): 223-237.
[8]	ZHOU Yu, XIE Wei, Kwong Tak Wu, JIANG Jianmin. Reconstruction of Video Snapshot Compressive Imaging Based on Triple Self-Attention [J]. Computer Engineering, 2025, 51(1): 20-30.
[9]	HU Shenglong, CHEN Bin, ZHANG Kaihua, SONG Huihui. Co-Saliency Object Detection Enhanced by Scene Structure Knowledge [J]. Computer Engineering, 2025, 51(1): 31-41.
[10]	YU Yongtao, SUN Ao, LI Ang, ZHU Linlin. Optimization Method for Classifier Output Repeatability Based on Siamese Networks [J]. Computer Engineering, 2025, 51(1): 118-127.
[11]	ZHANG Huiying, SHENG Wenshun. Improved Algorithm for Facial Age Recognition Based on Label Adaptation [J]. Computer Engineering, 2025, 51(1): 174-181.
[12]	YANG Hongju, JI Chang. Research on Learning-Driven Image Compression Algorithm [J]. Computer Engineering, 2025, 51(1): 190-197.
[13]	WANG Xiaolu, WEN Jianrong. Human Action Recognition Method Based on Action-Time Perception [J]. Computer Engineering, 2025, 51(1): 216-224.
[14]	HUO Jiuyuan, SU Hongrui, WU Zeyu, WANG Tingjuan. Road Traffic Small Target Vehicle Detection Algorithm Based on Improved YOLOv8 [J]. Computer Engineering, 2025, 51(1): 246-257.
[15]	WANG Qian, ZHANG Junhua, WANG Zetong, LI Bo. X2S-Net: Three-Dimensional Reconstruction of Spine Based on Biplanar X-Rays [J]. Computer Engineering, 2025, 51(1): 277-286.

Please choose a citation manager

Content to export