基于深度学习的交互笔关键点估计研究

doi:10.19678/j.issn.1000-3428.0066469

计算机工程 ›› 2023, Vol. 49 ›› Issue (10): 22-30. doi: 10.19678/j.issn.1000-3428.0066469

基于深度学习的交互笔关键点估计研究

朱兴帅¹^,², 叶彬¹^,², 姚康¹^,², 丁上上¹^,², 徐道亮¹^,², 付威威¹^,²^,*

1. 中国科学技术大学生物医学工程学院(苏州), 江苏苏州 215000
2. 中国科学院苏州生物医学工程技术研究所, 江苏苏州 215000

收稿日期:2022-12-08 出版日期:2023-10-15 发布日期:2023-10-10
通讯作者: 付威威
作者简介:
朱兴帅(1997-), 男, 硕士研究生, 主研方向为计算机视觉、图像处理
叶彬, 硕士研究生
姚康, 硕士研究生
丁上上, 硕士研究生
徐道亮, 博士研究生
基金资助:
中国科学院青年创新促进会项目(E1290301)

Research on Key Point Estimation of Interactive Pen Based on Deep Learning

Xingshuai ZHU¹^,², Bin YE¹^,², Kang YAO¹^,², Shangshang DING¹^,², Daoliang XU¹^,², Weiwei FU¹^,²^,*

1. School of Biomedical Engineering(Suzhou), University of Science and Technology of China, Suzhou 215000, Jiangsu, China
2. Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215000, Jiangsu, China

Received:2022-12-08 Online:2023-10-15 Published:2023-10-10
Contact: Weiwei FU

摘要/Abstract

摘要：

虚拟现实技术应用领域广泛，但现有交互方式不能满足使用者精细化操作的需求。通过交互笔可实现三维空间的精确输入，提升生产力效率。设计基于单目RGB图片的两阶段交互笔关键点估计模型PKPD-Net。通过CBAM-SHN网络得到二维关键点信息，利用笔的二维姿态特征进一步回归出关键点三维位置信息。该模型使用CBAM模块改进融合方式、基于Offset的关键点亚像素定位、辅助手部关键点预测等方法，实现高精度的笔上关键点三维估计，为通过交互笔进行精细化操作提供准确的位置信息。在大规模数据集上进行实验和验证，结果表明，相较于Minimal-hand与HOPE-Net模型，该模型预测关键点的mean_EPE分别降低0.882和0.710 mm，PSF@4分别提升31.38和32.31个百分点。最后，为探索产业级应用，结合PKPD-Net进行应用开发，通过时序关联实现操作轨迹的复原。

关键词: 虚拟现实, 精细操作, 深度学习, 关键点估计, 特征融合

Abstract:

Virtual Reality(VR) is widely used in various fields. However, the need for refined operations can not be met when using existing interaction methods. Using an interactive pen can achieve an accurate input of 3D space and enhance productivity efficiency. Thus, this study proposes a two-stage key point estimation algorithm of interactive pen based on a single RGB picture, PKPD-Net. In particular, the 2D key points are first estimated using the Convolutional Block Attention Module-Stacked Hourglass Network(CBAM-SHN). Then, the location information of 3D key points is calculated based on the 2D posture characteristics. This model proposes an improved fusion method based on the CBAM modules, sub-pixel positioning of key points based on Offset, and auxiliary estimation through the key points of supplementary hands. As a result, it achieves a highly precise estimation of the 3D key points. This provides accurate location information for refined operations through interactive pens. The model training and testing are performed on numerous datasets.The PKPD-Net achieves a mean End Point Error(mean EPE) of the key points that is lower by 0.882 and 0.710 mm, compared to that obtained using Minimal-hand and HOPE-Net models, respectively. Moreover, the Percentage of Success Frame with less than 4 mm(PSF@4) of key points achieved using the proposed model is higher by 31.38 and 32.31 percentage points, respectively. Thus, the method proposed in this study proves to be more advanced and effective than the existing methods. Finally, to explore the product applications, PKPD-Net is used to recover the operating trajectory through a time-sequential association.

Key words: Virtual Reality(VR), refined operation, deep learning, key point estimation, feature fusion

朱兴帅, 叶彬, 姚康, 丁上上, 徐道亮, 付威威. 基于深度学习的交互笔关键点估计研究[J]. 计算机工程, 2023, 49(10): 22-30.

Xingshuai ZHU, Bin YE, Kang YAO, Shangshang DING, Daoliang XU, Weiwei FU. Research on Key Point Estimation of Interactive Pen Based on Deep Learning[J]. Computer Engineering, 2023, 49(10): 22-30.

https://www.ecice06.com/CN/Y2023/V49/I10/22

图/表 14

图1 PKPD-Net网络结构

Fig.1 PKPD-Net network structure

图2 基于Offset偏移量的关键点亚像素定位

Fig.2 Sub-pixel positioning of key point based on Offset

图3 Hourglass Module网络结构

Fig.3 Hourglass Module network structure

图4 CBAM网络结构

Fig.4 CBAM network structure

图5 CBAM-SHN网络结构

Fig.5 CBAM-SHN network structure

图6 三维关键点估计网络结构

Fig.6 3D key point estimation network structure

图7 23个关键点标注规则

Fig.7 23 key points marking rules

图8 可视化估计结果对比

Fig.8 Comparison of visual estimation results

图9 书写实验示意图

Fig.9 Schematic diagrams of writing experiment

参考文献 27

1	杨青, 钟书华. 国外"虚拟现实技术发展及演化趋势"研究综述. 自然辩证法通讯, 2021, 43 (3): 97- 106. URL
	YANG Q , ZHONG S H . A review of foreign countries on the development and evolution trends of virtual reality technology. Journal of Dialectics of Nature, 2021, 43 (3): 97- 106. URL
2	KIRSH D . Thinking with external representations. AI & Society, 2010, 25 (4): 441- 454.
3	李柯泉, 陈燕, 刘佳晨, 等. 基于深度学习的目标检测算法综述. 计算机工程, 2022, 48 (7): 1- 12. URL
	LI K Q , CHEN Y , LIU J C , et al. Survey of deep learning-based object detection algorithms. Computer Engineering, 2022, 48 (7): 1- 12. URL
4	HE C H, LI R H, LI S, et al. Voxel set transformer: a set-to-set approach to 3D object detection from point clouds[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2022: 8407-8417.
5	YANG L H, ZHUO W, QI L, et al. ST: make self-training work better for semi-supervised semantic segmentation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2022: 4258-4267.
6	CHEN Z Z, WANG T, WU X W, et al. Class re-activation maps for weakly-supervised semantic segmentation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2022: 959-968.
7	李杰. 基于深度卷积神经网络的关键点检测算法研究. 成都: 中国科学院光电技术研究所, 2022, URL
	LI J . Research on keypoint detection algorithm based on deep convolutional neural network. Chengdu: Institute of Optics and Electronics, Chinese Academy of Sciences, 2022, URL
8	WANG Z T, NIE X C, QU X C, et al. Distribution-aware single-stage models for multi-person 3D pose estimation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2022: 13086-13095.
9	PAVLAKOS G, ZHOU X W, DERPANIS K G, et al. Coarse-to-fine volumetric prediction for single-image 3D human pose[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2017: 1263-1272.
10	SUN X, XIAO B, WEI F Y, et al. Integral human pose regression[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 536-553.
11	ZHAN Y, LI F H, WENG R L, et al. Ray3D: ray-based 3D human pose estimation for monocular absolute 3D localization[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2022: 13106-13115.
12	CHEN Y J, TU Z G, KANG D, et al. Model-based 3D hand reconstruction via self-supervised learning[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2021: 10446-10455.
13	LIU S W, JIANG H W, XU J R, et al. Semi-supervised 3D hand-object poses estimation with interactions in time[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2021: 14682-14692.
14	MARTINEZ J, HOSSAIN R, ROMERO J, et al. A simple yet effective baseline for 3d human pose estimation[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2017: 2659-2668.
15	LI C, LEE G H. Generating multiple hypotheses for 3D human pose estimation with mixture density network[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2020: 9879-9887.
16	CHEN C H, RAMANAN D. 3D human pose estimation=2D pose estimation matching[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2017: 5759-5767.
17	ZIMMERMANN C, BROX T. Learning to estimate 3D hand pose from single RGB images[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2017: 4913-4921.
18	MUELLER F, BERNARD F, SOTNYCHENKO O, et al. GANerated hands for real-time 3D hand tracking from monocular RGB[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2018: 49-59.
19	ZHOU K, HAN X G, JIANG N J, et al. HEMlets pose: learning part-centric heatmap triplets for accurate 3D human pose estimation[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2020: 2344-2353.
20	DOOSTI B, NAHA S, MIRBAGHERI M, et al. HOPE-Net: a graph-based model for hand-object pose estimation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2020: 6607-6616.
21	NEWELL A, YANG K Y, DENG J. Stacked hourglass networks for human pose estimation[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 483-499.
22	SZEGEDY C, IOFFE S, VANHOUCKE V, et al. Inception-v4, Inception-ResNet and the impact of residual connections on learning[EB/OL]. [2022-10-02]. https://arxiv.org/abs/1602.07261.
23	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2016: 770-778.
24	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 3-19.
25	黄青丹, 何彬彬, 宋浩永, 等. 基于双目立体视觉的目标空间坐标计算及姿态估计. 华南师范大学学报(自然科学版), 2020, 52 (2): 9- 13. URL
	HUANG Q D , HE B B , SONG H Y , et al. Space coordinate calculation and attitude estimation based on binocular stereo vision. Journal of South China Normal University(Natural Science Edition), 2020, 52 (2): 9- 13. URL
26	KINGMA D P, BA J. Adam: a method for stochastic optimization[EB/OL]. [2022-10-02]. https://arxiv.org/abs/1412.6980.
27	ZHOU Y X, HABERMANN M, XU W P, et al. Monocular real-time hand shape and motion capture using multi-modal data[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2020: 5345-5354.

[1]	谭巨全, 王然. 特征融合下田径录像3D人体动作DTW捕捉算法[J]. 计算机工程, 2024, 50(7): 71-78.
[2]	牛瑞婷, 严天峰, 高锐, 王映植. 低信噪比下基于深度学习TCNN-MobileNet的调制识别[J]. 计算机工程, 2024, 50(7): 204-215.
[3]	张溢文, 蔡满春, 陈咏豪, 朱懿, 姚利峰. 融合空间特征的多尺度深度伪造检测方法[J]. 计算机工程, 2024, 50(7): 240-250.
[4]	肖慈, 徐杨, 张永丹, 冯明文, 黄易仟. 结合注意力和低光增强的夜间语义分割[J]. 计算机工程, 2024, 50(7): 271-281.
[5]	张诗婧, 莫绪涛, 赵行, 董杨林. 基于球面折反射成像和YOLOv7的内螺纹缺陷检测[J]. 计算机工程, 2024, 50(7): 282-292.
[6]	王晋涛, 秦昂, 张元, 陈一飞, 王廷凤, 谢承霖, 邹刚. 基于注意力增强与特征融合的中文医学实体识别[J]. 计算机工程, 2024, 50(7): 324-332.
[7]	徐明亮, 李芳媛, 马浩然, 何飞. 大规模神经记录的峰电位聚类算法(特邀)[J]. 计算机工程, 2024, 50(6): 1-34.
[8]	魏琢艺, 罗迈, 李文兵, 曾远松, 余伟江, 杨跃东. 基于多源域适应的单细胞智能分类[J]. 计算机工程, 2024, 50(6): 48-55.
[9]	李亚康, 陈刚. 小角中子散射物理模型自动化筛选[J]. 计算机工程, 2024, 50(6): 56-64.
[10]	李子杰, 周菊香, 韩晓瑜, 甘健侯, 鹿泽光, 王俊. 序列特征与学习过程融合的知识追踪模型[J]. 计算机工程, 2024, 50(6): 77-85.
[11]	李永飞, 李铭洋, 常鑫, 曹可欣. 基于可解释性深度学习的物联网水质监测数据异常检测[J]. 计算机工程, 2024, 50(6): 179-187.
[12]	杨硕, 王一丁. 基于改进薄板样条运动模型的人脸动画算法[J]. 计算机工程, 2024, 50(6): 255-265.
[13]	李致金, 汤佳辉, 闫金凤. 基于边缘计算的轻量化识别方法[J]. 计算机工程, 2024, 50(6): 287-295.
[14]	李雪, 王雅文, 张前进. 基于信息检索的源代码自动命名[J]. 计算机工程, 2024, 50(6): 304-310.
[15]	梁松林, 林伟, 王珏, 杨庆. 面向后渗透攻击行为的网络恶意流量检测研究[J]. 计算机工程, 2024, 50(5): 128-138.

选择文件类型/文献管理软件名称

选择包含的内容

基于深度学习的交互笔关键点估计研究

Research on Key Point Estimation of Interactive Pen Based on Deep Learning

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 27

相关文章 15

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于深度学习的交互笔关键点估计研究

Research on Key Point Estimation of Interactive Pen Based on Deep Learning

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 27

相关文章 15

编辑推荐

Metrics

本文评价