基于姿态估计与GRU网络的人体康复动作识别

doi:10.19678/j.issn.1000-3428.0058201

摘要/Abstract

摘要： 康复锻炼是脑卒中患者的重要治疗方式，为提高康复动作识别的准确率与实时性，更好地辅助患者在居家环境中进行长期康复训练，结合姿态估计与门控循环单元（GRU）网络提出一种人体康复动作识别算法Pose-AMGRU。采用OpenPose姿态估计方法从视频帧中提取骨架关节点，经过姿态数据预处理后得到表达肢体运动的关键动作特征，并利用注意力机制构建融合三层时序特征的GRU网络实现人体康复动作分类。实验结果表明，该算法在KTH和康复动作数据集中的识别准确率分别为98.14%和100%，且在GTX1060显卡上的运行速度达到14.23 frame/s，具有较高的识别准确率与实时性。

关键词: 康复训练, 动作识别, 姿态估计, 门控循环单元, 注意力机制

Abstract: Rehabilitation exercise is an important treatment method for stroke patients.This paper proposes a rehabilitation action recognition algorithm,Pose-AMGRU,which combines pose estimation with Gated Recurrent Unit(GRU) in order to improve the accuracy and real-time performance of rehabilitation action recognition,and thus assist patients in in-home long-term rehabilitation training.The algorithm uses OpenPose pose estimation method to extract the skeleton joints from video frames,and the pose data is preprocessed to obtain the key action features that represent body movement.Then a GRU network with three-layer time series features is constructed by using the attention mechanism to realize rehabilitation action classification.Experimental results on KTH dataset and rehabilitation action dataset show that the proposed algorithm increases the recognition accuracy to 98.14% and 100%,and its running speed on GTX1060 reaches 14.23 frame/s,which demonstrates its excellent recognition accuracy and real-time performance.

Key words: rehabilitation training, action recognition, pose estimation, Gated Recurrent Unit(GRU), attention mechanism

中图分类号:

TP391.41

闫航, 陈刚, 佟瑶, 姬波, 胡北辰. 基于姿态估计与GRU网络的人体康复动作识别[J]. 计算机工程, 2021, 47(1): 12-20.

YAN Hang, CHEN Gang, TONG Yao, JI Bo, HU Beichen. Human Rehabilitation Action Recognition Based on Pose Estimation and GRU Network[J]. Computer Engineering, 2021, 47(1): 12-20.

https://www.ecice06.com/CN/Y2021/V47/I1/12

图/表 17

20210125162833

20210125162836

20210125162839

20210125162842

20210125162845

20210125162849

20210125162852

20210125162855

20210125162900

20210125162903

20210125162907

20210125162910

20210125162915

20210125162919

20210125162924

20210125162929

20210125162932

参考文献

[1] GAN Yong,YANG Tingting,LIU Jianxin,et al.Research progress on the prevalence trend and influencing factors of stroke at home and abroad[J].Chinese Preventive Medicine,2019,20(2):139-144.(in Chinese)甘勇,杨婷婷,刘建新,等.国内外脑卒中流行趋势及影响因素研究进展[J].中国预防医学杂志,2019,20(2):139-144.
[2] DAUNORAVICIENE K,ADOMAVICIENE A.Effects of robot-assisted training on upper limb functional recovery during the rehabilitation of poststroke patients[J].Technology and Health Care Official Journal of the European Society for Engineering and Medicine,2018,26(S2):1-10.
[3] SHENG Han,SHAO Shengwen,WANG Huiqin,et al.A study on the rehabilitation exercise compliance curve in stroke patients[J].Chinese Journal of Nursing,2016,51(6):712-715.(in Chinese)盛晗,邵圣文,王惠琴,等.脑卒中患者康复锻炼依从性动态变化的研究[J].中华护理杂志,2016,51(6):712-715.
[4] BISIO I,DELFINO A,LAVAGETTO F,et al.Enabling IoT for in-home rehabilitation:accelerometer signals classification methods for activity and movement recognition[J].IEEE Internet of Things Journal,2016,4(1):135-146.
[5] AHMADI M,O'NEIL M,FRAGALA-PINKHAM M,et al.Machine learning algorithms for activity recognition in ambulant children and adolescents with cerebral palsy[J].Journal of NeuroEngineering and Rehabilitation,2018,15(1):105-114.
[6] ZUO Guoyu,XU Zhaokun,LU Jiahao,et al.A structure-optimized DDAG-SVM action recognition method for upper limb rehabilitation training[J].Acta Automatica Sinica,2020,46(3):549-561.(in Chinese)左国玉,徐兆坤,卢佳豪,等.基于结构优化的DDAG-SVM上肢康复训练动作识别方法[J].自动化学报,2020,46(3):549-561.
[7] PEI Xiaomin,FAN Huijie,TANG Yandong.Action recognition method of spatio-temporal feature fusion deep learning network[J].Infrared and Laser Engineering,2018,47(2):46-51.(in Chinese)裴晓敏,范慧杰,唐延东.时空特征融合深度学习网络人体行为识别方法[J].红外与激光工程,2018,47(2):46-51.
[8] XIAO Xiao,XU Dan,WAN Wanggen.Overview:video recognition from handcrafted method to deep learning method[C]//Proceedings of International Conference on Audio,Language and Image.Washington D.C.,USA:IEEE Press,2017:646-651.
[9] JI Shuiwang,XU Wei,YANG Ming,et al.3D convolutional neural networks for human action recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2013,35(1):221-231.
[10] DONAHUE J,HENDRICKS L A,ROHRBACH M,et al.Long-term recurrent convolutional networks for visual recognition and description[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2014,36(4):677-691.
[11] SIMONYAN K,ZISSERMAN A.Two-stream convolutional networks for action recognition in videos[J].Advances in Neural Information Processing Systems,2014,1:568-576.
[12] CAO Congqi,ZHANG Yifan,ZHANG Chunjie,et al.Body joint guided 3-D deep convolutional descriptors for action recognition[J].IEEE Transactions on Cybernetics,2018,48(3):1095-1108.
[13] FONTES D G A E,MENEZES C T D,PASCAL F,et al.Rehabilitation motion recognition based on the international biomechanical standards[J].Expert Systems with Application,2019,116:396-409.
[14] HBALI Y,HBALI S,BALLIHI L,et al.Skeleton-based human activity recognition for elderly monitoring systems[J].IET Computer Vision,2018,12(1):16-26.
[15] ZHANG Qingzhi,WU Panfeng,DU Xiaohui,et al.Rehabilitation recognition skeleton data depth learning based on RNN[EB/OL].[2020-03-14].http://doi.org/10.1051/matecconf/201927702007.
[16] CAO Z,HIDALGO G,SIMON T,et al.OpenPose:realtime multi-person 2D pose estimation using part affinity fields[EB/OL].[2020-03-14].https://doi.org/10.1109/TPAMI.2019.2929257.
[17] CHO K,VAN M B,BAHDANAU D,et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[EB/OL].[2020-03-14].https://arxiv.org/abs/1406.1078.
[18] LIU W,ANGUELOV D,ERHAN D,et al.SSD:single shot multibox detector[C]//Proceedings of European Conference on Computer Vision.Berlin,Germany:Springer,2016:21-37.
[19] XU Mengya,YANG Weimin.Application of family medical gymnastics in community rehabilitation of ischemic stroke[J].Chinese Journal of Gerontology,2010,30(17):41-42.(in Chinese)许梦雅,杨伟民.家庭医疗体操在缺血性脑卒中社区康复中的应用[J].中国老年学杂志,2010,30(17):41-42.
[20] SCHULDT C,LAPTEV I,CAPUTO B.Recognizing human actions:a local SVM approach[C]//Proceedings of the 17th International Conference on Pattern Recognition.Washington D.C.,USA:IEEE Press,2004:32-36.
[21] MEGRHI S,JMAL M,SOUIDENE W,et al.Spatio-temporal action localization and detection for human action recognition in big dataset[J].Journal of Visual Communication and Image Representation,2016,41:375-390.
[22] LU Tianran,YU Fengqin,CHEN Ying.A human action recognition method based on LSDA dimension reduction[J].Computer Engineering,2019,45(3):237-241,249.(in Chinese)鹿天然,于凤芹,陈莹.一种基于线性序列差异分析降维的人体行为识别方法[J].计算机工程,2019,45(3):237-241,249.
[23] MA Yuxi,TAN Li,DONG Xu,et al.Action recognition for intelligent monitoring[J].Journal of Image and Graphics,2019,24(2):128-136.(in Chinese)马钰锡,谭励,董旭,等.面向智能监控的行为识别[J].中国图象图形学报,2019,24(2):128-136.
[24] SARGANO A B,WANG X,ANGELOV P,et al.Human action recognition using transfer learning with deep representations[C]//Proceedings of 2017 International Joint Conference on Neural Networks.Washington D.C.,USA:IEEE Press,2017:463-469.
[25] LI S,LI W Q,COOK C,et al.Independently Recurrent Neural Network(IndRNN):building a longer and deeper RNN[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:5457-5466.
[26] LEI T,ZHANG Yu,WANG S I,et al.Simple recurrent units for highly parallelizable recurrence[C]//Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing.Philadelphia,USA:ACL Press,2018:4470-4481.
[27] TRAN D,BOURDEV L,FERGUS R,et al.Learning spatio-temporal features with 3D convolutional networks[C]//Proceedings of 2015 IEEE International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2015:4489-4497.

选择文件类型/文献管理软件名称

选择包含的内容