基于单-多视图优化的足球球员三维姿态和体型估计

doi:10.19678/j.issn.1000-3428.0067480

摘要/Abstract

摘要：

足球比赛场景的三维重建有助于观众自由切换视角，增加了互动性和沉浸感。针对足球比赛场景中的足球球员，提出一种三维姿态和体型估计方法。对球员的多视图图像使用训练好的部分注意力回归的三维人体估计(PARE)模型生成初始的三维姿态和体型估计，并使用人工标注的二维关节点作为优化目标。单-多视图优化操作利用蒙皮多人线性模型(SMPL)和正交投影的可微性，将球员的三维姿态和体型参数映射到二维关节点，计算其与人工标注之间的差异，再使用神经网络的反向传播算法更新三维姿态和体型参数，持续这些过程直到差异最小化。在自建的足球球员多视图数据集上的实验结果表明，该方法能够有效估计足球球员的三维姿态和体型，与人体网格恢复、在循环中优化SMPL、PARE等方法相比，二维关节点精度在单视图上提高了9.2%~37.5%，在多视图交叉验证中提高了34.9%~54.1%。

关键词: 三维姿态和体型估计, 参数化人体模型, 单-多视图优化, 反向传播, 蒙皮多人线性模型

Abstract:

Reconstructing a Three-Dimensional(3D) scene of a soccer game allows the audience to switch their viewpoints freely, enhancing interactivity and immersion. A pipeline to estimate the 3D pose and shape of soccer players in a soccer scene is proposed. The proposed method obtains an initial 3D pose and shape estimation of soccer players with a trained Part Attention Regressor for a 3D human body Estimation(PARE) model, and utilizes annotated Two-Dimensional(2D) joints of the human body as the optimization target. Single- and multi-view optimization operations utilize the differentiability of a Skinned Multi-Person Linear(SMPL) model and orthogonal projection, map the 3D pose and shape parameters to predicted 2D joints, obtain the difference with annotations, and subsequently update these parameters with a backpropagation algorithm of neural networks. These processes continue until the difference is minimized. Experimental results on our soccer players multi-view dataset demonstrate that the proposed method is able to estimate the 3D pose and body shape of soccer players. Compared with Human Mesh Recovery(HMR), SMPL oPtimization IN the loop(SPIN), and PARE methods, the accuracy of 2D joints improved by 9.2%-37.5% on single-view, and by 34.9%-54.1% on multi-view cross validation.

Key words: Three-Dimensional(3D) pose and body shape estimation, parametric human body model, single- and multi-view optimization, backpropagation, Skinned Multi-Person Linear(SMPL) model

谢欢, 刘纯平, 季怡. 基于单-多视图优化的足球球员三维姿态和体型估计[J]. 计算机工程, 2024, 50(3): 200-207.

Huan XIE, Chunping LIU, Yi JI. Three-Dimensional Pose and Body Shape Estimation of Soccer Players Based on Single- and Multi-View Optimization[J]. Computer Engineering, 2024, 50(3): 200-207.

http://www.ecice06.com/CN/Y2024/V50/I3/200

图/表 10

图1 足球球员三维姿态和体型估计总体框架

Fig.1 Overall framework of 3D pose and body shape estimation of soccer players

图2 人体二维关节点示例

Fig.2 Example of human body 2D joint points

图3 足球游戏《FIFA 21》中的不同视图

Fig.3 Different views in the soccer game FIFA 21

图4 二维关节点检测结果

Fig.4 2D joint point detection results

图5 三维姿态和体型的初始估计误差

Fig.5 Initial estimation error of 3D pose and body shape

图6 单视图优化结果应用到其他视图上的差异比较

Fig.6 Comparison of differences by applying single-view optimization results to other views

图7 所提方法与其他方法的可视化结果比较

Fig.7 Comparison of visualization results of the proposed method with other methods

参考文献 37

1	VAROL G, CEYLAN D, RUSSELL B, et al. BodyNet: volumetric inference of 3D human body shapes. Berlin, Germany: Springer, 2018.
2	万云翀, 宋云鹏, 刘利刚. 基于体素联合坐标的单人三维姿态估计. 计算机辅助设计与图形学学报, 2022, 34(9): 1411- 1419. URL
	WAN Y C, SONG Y P, LIU L G. 3D human pose estimation based on volumetric joint coordinates. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(9): 1411- 1419. URL
3	KOLOTOUROS N, PAVLAKOS G, DANIILIDIS K. Convolutional mesh regression for single-image human shape reconstruction[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 4496-4505.
4	LIN K, WANG L J, LIU Z C. End-to-end human pose and mesh reconstruction with transformers[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 1954-1963.
5	ANGUELOV D, SRINIVASAN P, KOLLER D, et al. SCAPE: shape completion and animation of people. ACM Transactions on Graphics, 2005, 24(3): 408- 416. doi: 10.1145/1073204.1073207
6	JAIN A, THORMÄHLEN T, SEIDEL H P, et al. MovieReshape: tracking and reshaping of humans in videos. ACM Transactions on Graphics, 2010, 29(6): 148.
7	TSOLI A, MAHMOOD N, BLACK M J. Breathing life into shape. ACM Transactions on Graphics, 2015, 33(4): 92- 100.
8	LOPER M, MAHMOOD N, ROMERO J, et al. SMPL: a skinned multi-person linear model. ACM Transactions on Graphics, 2015, 34(6): 248.
9	PAVLAKOS G, CHOUTAS V, GHORBANI N, et al. Expressive body capture: 3D hands, face, and body from a single image[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 10975-10985.
10	曾志超, 李桂清, 邹歆仪, 等. 三维人体模型姿态与形状重构. 计算机辅助设计与图形学学报, 2019, 31(9): 1485- 1493. URL
	ZENG Z C, LI G Q, ZOU X Y, et al. Human model pose and shape reconstruction. Journal of Computer-Aided Design & Computer Graphics, 2019, 31(9): 1485- 1493. URL
11	李健, 杨镖镖, 张皓若. 基于SMPL模型的人体姿态估计. 计算机仿真, 2021, 38(3): 292-297, 486. doi: 10.3969/j.issn.1006-9348.2021.03.060
	LI J, YANG B B, ZHANG H R. Human pose estimation based on SMPL model. Computer Simulation, 2021, 38(3): 292-297, 486. doi: 10.3969/j.issn.1006-9348.2021.03.060
12	LI J F, XU C, CHEN Z C, et al. HybrIK: a hybrid analytical-neural inverse kinematics solution for 3D human pose and shape estimation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 3382-3392.
13	KANAZAWA A, BLACK M J, JACOBS D W, et al. End-to-end recovery of human shape and pose[EB/OL]. [2023-03-07]. https://arxiv.org/abs/1712.06584.
14	KOLOTOUROS N, PAVLAKOS G, BLACK M, et al. Learning to reconstruct 3D human pose and shape via model-fitting in the loop[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 2252-2261.
15	CHUN S, PARK S, CHANG J Y. Learnable human mesh triangulation for 3D human pose and shape estimation[C]//Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision. Washington D. C., USA: IEEE Press, 2023: 2849-2858.
16	YU X, VAN BAAR J, CHEN S H. Joint 3D human shape recovery and pose estimation from a single image with bilayer graph[C]//Proceedings of International Conference on 3D Vision. Washington D. C., USA: IEEE Press, 2021: 505-514.
17	RONG Y, LIU Z W, LI C, et al. Delving deep into hybrid annotations for 3D human recovery in the wild[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 5339-5347.
18	JENA S, MULTON F, BOUKHAYMA A. Monocular human shape and pose with dense mesh-borne local image features[C]//Proceedings of the 16th IEEE International Conference on Automatic Face and Gesture Recognition. New York, USA: ACM Press, 2021: 1-5.
19	TAN V, BUDVYTIS I, CIPOLLA R. Indirect deep structured learning for 3D human body shape and pose prediction[C]//Proceedings of 2017 British Machine Vision Conference. London, UK: British Machine Vision Association, 2017: 10753607.
20	LI Z G, HEYDEN A, OSKARSSON M. A novel joint points and silhouette-based method to estimate 3D human pose and shape. International Journal of Computer Graphics, 2023, 39(5): 1893- 1906.
21	OMRAN M, LASSNER C, PONS-MOLL G, et al. Neural body fitting: unifying deep learning and model based human pose and shape estimation[EB/OL]. [2023-03-07]. https://arxiv.org/abs/1808.05942.
22	KOCABAS M, HUANG C H P, HILLIGES O, et al. PARE: part attention regressor for 3D human body estimation[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 11107-11117.
23	ZHANG H W, CAO J, LU G, et al. DaNet: decompose-and-aggregate network for 3D human shape and pose estimation[C]//Proceedings of the 27th ACM International Conference on Multimedia. New York, USA: ACM Press, 2019: 935-944.
24	SENGUPTA A, BUDVYTIS I, CIPOLLA R. Synthetic training for accurate 3D human pose and shape estimation in the wild[EB/OL]. [2023-03-07]. https://arxiv.org/abs/2009.10013.
25	LI Z G, OSKARSSON M, HEYDEN A. 3D human pose and shape estimation through collaborative learning and multi-view model-fitting[C]//Proceedings of IEEE Winter Conference on Applications of Computer Vision. Washington D. C., USA: IEEE Press, 2021: 1888-1897.
26	LASSNER C, ROMERO J, KIEFEL M, et al. Unite the people: closing the loop between 3D and 2D human representations[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 4704-4713.
27	DONG Z J, SONG J, CHEN X, et al. Shape-aware multi-person pose estimation from multi-view images[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 11138-11148.
28	王伟楠, 张荣, 郭立君. 结合稀疏表示和深度学习的视频中3D人体姿态估计. 中国图象图形学报, 2020, 25(3): 456- 467. URL
	WANG W N, ZHANG R, GUO L J. Video based 3D human pose estimation combining sparse representation and deep learning. Journal of Image and Graphics, 2020, 25(3): 456- 467. URL
29	IONESCU C, PAPAVA D, OLARU V, et al. Human3.6M: large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(7): 1325- 1339.
30	KASAEI S, FOTOUHI M. Azadi soccer dataset[EB/OL]. [2023-03-07]. http://ipl.ce.sharif.edu/azadi_soccer_dataset.html.
31	REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137- 1149. doi: 10.1109/TPAMI.2016.2577031
32	WU Y X, KIRILLOV A, MASSA F, et al. Detectron2[EB/OL]. [2023-03-07]. https://github.com/facebookresearch/detectron2.
33	RICHTER S R, VINEET V, ROTH S, et al. Playing for data: ground truth from computer games. Berlin, Germany: Springer, 2016.
34	CALAGARI K, ELGHARIB M, DIDYK P, et al. Data driven 2D-to-3D video conversion for soccer. IEEE Transactions on Multimedia, 2018, 20(3): 605- 619.
35	REMATAS K, KEMELMACHER-SHLIZERMAN I, CURLESS B, et al. Soccer on your tabletop[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 4738-4747.
36	KINGMA D, BA J. Adam: a method for stochastic optimization[EB/OL]. [2023-03-07]. https://arxiv.org/abs/1412.6980.
37	ZHOU Y, BARNES C, LU J W, et al. On the continuity of rotation representations in neural networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 5738-5746.

[1]	尚瑛杰, 董丽亚, 何虎. 基于脉冲神经网络的迁移学习算法与软件框架[J]. 计算机工程, 2020, 46(3): 53-59.
[2]	周天祥, 董红斌, 周雯. 基于区域分割的混合地震预测算法[J]. 计算机工程, 2019, 45(9): 310-315.
[3]	胡翩翩,曾碧卿. 基于HS-BP神经网络的认知无线电频谱预测技术[J]. 计算机工程, 2017, 43(7): 146-150,155.
[4]	江帆,刘辉,王彬,孙晓峰,代照坤. 基于CNN-GRNN模型的图像识别[J]. 计算机工程, 2017, 43(4): 257-262.
[5]	刘卫明,高晓东,毛伊敏,周昭飞. 不确定遗传神经网络在滑坡危险性预测中的研究与应用[J]. 计算机工程, 2017, 43(2): 308-316.
[6]	王吉军,程华. 通用图形处理器功耗估算模型[J]. 计算机工程, 2017, 43(2): 92-97,104.
[7]	张宗华,赵京湘,卢享,牛新征. 基于遗传算法的BP神经网络在电力负载预测中的应用[J]. 计算机工程, 2017, 43(10): 277-282,288.
[8]	李炯城, 肖恒辉, 李桂愉. 神经网络中处理鞍点的LMBP改进算法[J]. 计算机工程, 2012, 38(23): 173-176,180.
[9]	王爱平, 江丽. 基于PSO的BP神经网络学习算法[J]. 计算机工程, 2012, 38(21): 193-196.
[10]	周媛, 宋海涛, 蒋砚军. 用于能耗数据分析的改进并行BP算法[J]. 计算机工程, 2012, 38(18): 171-173.
[11]	蒋行国, 褚福刚, 陈真诚, 梁晋涛. LAPS型电子舌神经网络味觉识别[J]. 计算机工程, 2012, 38(13): 26-29.
[12]	吕艳辉, 宫瑞敏. 计算机博弈中估值算法与博弈训练的研究[J]. 计算机工程, 2012, 38(11): 163-166.
[13]	王曦, 毕贵红, 唐京瑞. 基于EMD的太阳黑子时间序列组合预测模型[J]. 计算机工程, 2011, 37(24): 176-179.
[14]	黄鹏, 成怡. 改进的BP神经网络算法在航迹匹配中的应用[J]. 计算机工程, 2011, 37(11): 218-219,222.
[15]	王树, 杜启军, 余桂贤, 余生晨, 李广平, 徐亚飞3, 薛阳, 王晓伟. 网络入侵检测系统的最优特征选择方法[J]. 计算机工程, 2010, 36(15): 140-141,144.

选择文件类型/文献管理软件名称

选择包含的内容