基于注意力机制的人体关键点隐式建模网络

doi:10.19678/j.issn.1000-3428.0067134

摘要/Abstract

摘要：

人体姿态估计任务需要利用视觉线索和关节间的解剖关系来定位关键点，但基于卷积神经网络的方法难以关注远程上下文线索和建模远距离关节之间的依赖关系。为此，提出一种基于注意力机制的隐式建模方法，通过多阶段迭代计算关节之间的特征相关性来隐式建模关键点间的约束关系，消除卷积神经网络的局部操作，扩大网络的感受野，建模远距离关节之间的依赖关系。为了解决网络在训练过程中可能弱化不可见关键点的问题，采用焦点损失函数，使网络更关注于复杂的关键点。使用目前精度最高的特征提取高分辨率网络（HRNet）和经典特征提取残差网络（ResNet）作为主干网络进行实验，结果表明，在同等实验条件下，隐式建模方法可以提高人体姿态估计网络的性能，在MPII和MSCOCO人体姿态估计基准数据集上，以HRNet网络为主干网络的算法相较于原网络，精度分别提升了1.7%和2.6%。

关键词: 人体姿态估计, 卷积神经网络, 注意力机制, 焦点损失, 隐式建模

Abstract:

Human pose estimation necessitates the use of visual cues and anatomical joint relationships to pinpoint key points. Existing Convolutional Neural Network(CNN) methods falter in addressing long-range contextual cues and modeling dependencies among distant joints. This paper introduces an attention-based implicit modeling method that iteratively computes feature correlations between joints, thus implicitly modeling the constraint relationships among key points. This method diverges from the localized operations characteristic of CNN by expanding the network's receptive field and modeling dependencies between distantly positioned joints. To counteract the diminished visibility of crucial keypoints during network training, a focal loss function is implemented, prompting the network to concentrate on complex keypoints. Comparative experiments were performed under identical conditions using the state-of-the-art High-Resolution Network(HRNet) and the classic Residual Network(ResNet) as backbone networks. Results reveal that the implicit modeling network enhances human pose estimation performance. For instance, utilizing HRNet as the backbone, the algorithm's accuracy on the MPII and MSCOCO human pose estimation benchmark datasets improved by 1.7% and 2.6%, respectively, surpassing the original network's performance.

Key words: human pose estimation, convolutional neural network, attention mechanism, focal loss, implicit modeling

赵佳圆, 张玉茹, 苏晓东, 徐红岩, 李世洲, 张玉荣. 基于注意力机制的人体关键点隐式建模网络[J]. 计算机工程, 2024, 50(3): 317-325.

Jiayuan ZHAO, Yuru ZHANG, Xiaodong SU, Hongyan XU, Shizhou LI, Yurong ZHANG. Implicit Modeling Network of Human Keypoints Based on Attention Mechanism[J]. Computer Engineering, 2024, 50(3): 317-325.

http://www.ecice06.com/CN/Y2024/V50/I3/317

图/表 13

图1 CBAM注意力机制

Fig.1 CBAM attention mechanism

图2 十字交叉注意力机制

Fig.2 Cross attention mechanism

图3 关键点隐式建模架构

Fig.3 Architecture of key points implicit modeling

图4 HRNet网络结构

Fig.4 HRNet network structure

图5 多阶段关节隐式建模网络

Fig.5 Multistage joint implicit modeling network

图6 不同骨干网络的可视化结果

Fig.6 Visualization results of different backbone networks

图7 在MSCOCO和MPII验证集上的可视化结果

Fig.7 Visualization results on MSCOCO and MPII validation sets

参考文献 36

1	刘勇, 李杰, 张建林, 等. 基于深度学习的二维人体姿态估计研究进展. 计算机工程, 2021, 47(3): 1- 16. URL
	LIU Y, LI J, ZHANG J L, et al. Research progress of two-dimensional human pose estimation based on deep learning. Computer Engineering, 2021, 47(3): 1- 16. URL
2	GAO Q, LIU J G, JU Z J, et al. Dual-hand detection for human-robot interaction by a parallel network based on hand detection and body pose estimation. IEEE Transactions on Industrial Electronics, 2019, 66(12): 9663- 9672. doi: 10.1109/TIE.2019.2898624
3	AVOLA D, CINQUE L, FAGIOLI A, et al. 3D hand pose and shape estimation from RGB images for keypoint-based hand gesture recognition. Pattern Recognition, 2022, 129, 108762. doi: 10.1016/j.patcog.2022.108762
4	XIAO B, WU H P, WEI Y C. Simple baselines for human pose estimation and tracking[C]//Proceedings of the 15th European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 472-487.
5	NEWELL A, YANG K Y, DENG J. Stacked hourglass networks for human pose estimation[C]//Proceedings of the 14th European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 483-499.
6	YANG W, LI S, OUYANG W L, et al. Learning feature pyramids for human pose estimation[C]//Proceedings of the 16th IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 357-369.
7	SUN K, XIAO B, LIU D, et al. Deep high-resolution representation learning for human pose estimation[C]//Proceedings of the 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 578-586.
8	WANG X Y, TONG J W, WANG R. Attention refined network for human pose estimation. Neural Processing Letters, 2021, 53(4): 2853- 2872. doi: 10.1007/s11063-021-10523-9
9	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the 15th European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 3-19.
10	ZHANG H, GOODFELLOW I, METAXAS D N, et al. Self-attention generative adversarial networks[C]//Proceedings of the 36th International Conference on Machine Learning. Washington D. C., USA: IEEE Press, 2018: 256-266.
11	HUANG Z L, WANG X G, HUANG L C, et al. CCNet: criss-cross attention for semantic segmentation[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 652-667.
12	FELZENSZWALB P F, HUTTENLOCHER D P. Pictorial structures for object recognition. International Journal of Computer Vision, 2005, 61(1): 55- 79. doi: 10.1023/B:VISI.0000042934.15159.49
13	DANTONE M, GALL J, LEISTNER C, et al. Human pose estimation using body parts dependent joint regressors[C]//Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2013: 3041-3048.
14	TOSHEV A, SZEGEDY C. DeepPose: human pose estimation via deep neural networks[C]//Proceedings of the 27th IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2014: 453-466.
15	TOMPSON J, JAIN A, LECUN Y, et al. Joint training of a convolutional network and a graphical model for human pose estimation[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2014: 1799-1807.
16	CHEN Y L, WANG Z C, PENG Y X, et al. Cascaded pyramid network for multi-person pose estimation[C]//Proceedings of the 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 221-235.
17	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 770-778.
18	LIU S G, LI Y, HUA G G. Human pose estimation in video via structured space learning and halfway temporal evaluation. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 29(7): 2029- 2038. doi: 10.1109/TCSVT.2018.2858828
19	FENG R Y, GAO Y X, MA X Q, et al. Mutual information-based temporal difference learning for human pose estimation in video[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2023: 256-268.
20	SU X D, LIANG H Y, YAO G L, et al. Small-scale image semantic segmentation method based on multilevel superposition and enhancement fusion[C]//Proceedings of the 33rd Chinese Control and Decision Conference. Washington D. C., USA: IEEE Press, 2021: 366-375.
21	SU X, ZHANG Y, WANG C, et al. Multi-scale object detection algorithm based on faster R-CNN[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 2423-2435.
22	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[EB/OL]. [2023-02-05]. https://arxiv.org/abs/2010.11929.pdf.
23	ZHENG C, ZHU S J, MENDIETA M, et al. 3D human pose estimation with spatial and temporal transformers[C]//Proceedings of the 18th IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 641-654.
24	LI Y J, ZHANG S K, WANG Z C, et al. TokenPose: learning keypoint Tokens for human pose estimation[C]//Proceedings of the 18th IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 331-346.
25	YANG S, QUAN Z B, NIE M, et al. TransPose: keypoint localization via transformer[C]//Proceedings of the 18th IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 556-567.
26	DING X H, ZHANG X Y, HAN J G, et al. Scaling up your kernels to 31 × 31: revisiting large kernel design in CNNs[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 156-168.
27	ANDRILUKA M, PISHCHULIN L, GEHLER P, et al. 2D human pose estimation: new benchmark and state of the art analysis[C]//Proceedings of the 27th IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2014: 318-327.
28	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of the 13th European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 740-755.
29	YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence and the 30th Innovative Applications of Artificial Intelligence Conference and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence. New York, USA: ACM Press, 2018: 7444-7452.
30	WEI S E, RAMAKRISHNA V, KANADE T, et al. Convolutional pose machines[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 379-392.
31	PISHCHULIN L, INSAFUTDINOV E, TANG S Y, et al. DeepCut: joint subset partition and labeling for multi person pose estimation[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 332-345.
32	刘勇, 李杰, 任立成, 等. 并联化高分辨网络的人体姿态估计方法. 计算机工程与设计, 2022, 43(1): 237- 244. URL
	LIU Y, LI J, REN L C, et al. Human pose estimation based on parallel high-resolution net. Computer Engineering and Design, 2022, 43(1): 237- 244. URL
33	李坤, 侯庆. 基于注意力机制的轻量型人体姿态估计. 计算机应用, 2022, 42(8): 2407- 2414. URL
	LI K, HOU Q. Lightweight human pose estimation based on attention mechanism. Journal of Computer Applications, 2022, 42(8): 2407- 2414. URL
34	余保玲, 虞松坤, 孙耀然, 等. 基于DeepPose和Faster RCNN的多目标人体骨骼节点检测算法. 中国科学院大学学报, 2020, 37(6): 828- 834. URL
	YU B L, YU S K, SUN Y R, et al. Human body joint nodes detection based on DeepPose and Faster RCNN. Journal of University of Chinese Academy of Sciences, 2020, 37(6): 828- 834. URL
35	江春灵, 曾碧, 姚壮泽, 等. 融合权重自适应损失和注意力的人体姿态估计. 计算机工程与应用, 2023, 59(18): 145- 153. URL
	JIANG C L, ZENG B, YAO Z Z, et al. Human pose estimation fusing weight adaptive loss and attention. Computer Engineering and Applications, 2023, 59(18): 145- 153. URL
36	刘豪, 吴红兰, 房宇轩. 结合全局上下文信息的高效人体姿态估计. 计算机工程, 2023, 49(7): 102-109, 117. URL
	LIU H, WU H L, FANG Y X. Efficient human pose estimation combining global contextual information. Computer Engineering, 2023, 49(7): 102-109, 117. URL

[1]	王柏涵, 姜晓燕, 范柳伊. 基于深度监督隐空间构建的语义分割改进方法[J]. 计算机工程, 2024, 50(3): 191-199.
[2]	谢新林, 尹东旭, 张涛源, 谢刚. 基于注意力机制的多尺度融合人群计数算法[J]. 计算机工程, 2024, 50(3): 290-297.
[3]	袁文涛, 卫文韬, 高德民. 融合注意力机制的多视图卷积手势识别研究[J]. 计算机工程, 2024, 50(3): 208-215.
[4]	徐芳芯, 樊嵘, 马小陆. 面向拥挤行人检测的改进YOLOv7算法[J]. 计算机工程, 2024, 50(3): 250-258.
[5]	姜百浩, 刘静, 仇大伟, 姜良. 深度学习在脊柱图像分割中的应用综述[J]. 计算机工程, 2024, 50(3): 1-15.
[6]	赵博超, 马嘉骏, 崔磊, 栾文鹏, 朱静. 基于改进VMD-XGBoost-BiLSTM组合模型的光伏发电异常检测[J]. 计算机工程, 2024, 50(3): 306-316.
[7]	徐浩宸, 刘满华. 基于多层次自注意力网络的人脸特征点检测[J]. 计算机工程, 2024, 50(2): 239-246.
[8]	李倩, 魏伟波, 杨光宇, 宋金涛, 孙璐, 潘振宽. 基于先验驱动深度神经网络的泊松去噪变分模型[J]. 计算机工程, 2024, 50(2): 273-280.
[9]	刘帅威, 李智, 王国美, 张丽. 基于Transformer和GAN的对抗样本生成算法[J]. 计算机工程, 2024, 50(2): 180-187.
[10]	王正家, 胡飞飞, 张成娟, 雷卓, 何涛. 引入轻量级Transformer的自适应窗口立体匹配算法[J]. 计算机工程, 2024, 50(2): 256-265.
[11]	郭祥振, 李思潼, 卢锐, 郭森, 崔学荣, 杨钢. 基于多任务联合注意力的结肠息肉分割网络[J]. 计算机工程, 2024, 50(2): 327-336.
[12]	丁国辉, 刘宇琪, 王言开, 耿施展, 姜天昊. 基于翻转网络的低相关性序列数据预测研究[J]. 计算机工程, 2024, 50(2): 78-90.
[13]	圣文顺, 余熊峰, 林佳燕, 陈欣. 融合注意力与特征金字塔的小尺度目标检测算法[J]. 计算机工程, 2024, 50(1): 242-250.
[14]	吴志强, 解庆, 李琳, 刘永坚. 基于多模态融合的图神经网络推荐算法[J]. 计算机工程, 2024, 50(1): 91-100.
[15]	申秀雨, 姬伟峰, 李映岐, 吴玄. 面向边缘计算的TCA1C DDoS检测模型[J]. 计算机工程, 2024, 50(1): 198-205.

选择文件类型/文献管理软件名称

选择包含的内容