基于深度学习的交互笔关键点估计研究

doi:10.19678/j.issn.1000-3428.0066469

计算机工程 ›› 2023, Vol. 49 ›› Issue (10): 22-30. doi: 10.19678/j.issn.1000-3428.0066469

基于深度学习的交互笔关键点估计研究

朱兴帅¹^,², 叶彬¹^,², 姚康¹^,², 丁上上¹^,², 徐道亮¹^,², 付威威¹^,²^,*

1. 中国科学技术大学生物医学工程学院(苏州), 江苏苏州 215000
2. 中国科学院苏州生物医学工程技术研究所, 江苏苏州 215000

收稿日期:2022-12-08 出版日期:2023-10-15 发布日期:2023-10-10
通讯作者: 付威威
作者简介:
朱兴帅(1997-), 男, 硕士研究生, 主研方向为计算机视觉、图像处理
叶彬, 硕士研究生
姚康, 硕士研究生
丁上上, 硕士研究生
徐道亮, 博士研究生
基金资助:
中国科学院青年创新促进会项目(E1290301)

Research on Key Point Estimation of Interactive Pen Based on Deep Learning

Xingshuai ZHU¹^,², Bin YE¹^,², Kang YAO¹^,², Shangshang DING¹^,², Daoliang XU¹^,², Weiwei FU¹^,²^,*

1. School of Biomedical Engineering(Suzhou), University of Science and Technology of China, Suzhou 215000, Jiangsu, China
2. Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215000, Jiangsu, China

Received:2022-12-08 Online:2023-10-15 Published:2023-10-10
Contact: Weiwei FU

摘要/Abstract

摘要：

虚拟现实技术应用领域广泛，但现有交互方式不能满足使用者精细化操作的需求。通过交互笔可实现三维空间的精确输入，提升生产力效率。设计基于单目RGB图片的两阶段交互笔关键点估计模型PKPD-Net。通过CBAM-SHN网络得到二维关键点信息，利用笔的二维姿态特征进一步回归出关键点三维位置信息。该模型使用CBAM模块改进融合方式、基于Offset的关键点亚像素定位、辅助手部关键点预测等方法，实现高精度的笔上关键点三维估计，为通过交互笔进行精细化操作提供准确的位置信息。在大规模数据集上进行实验和验证，结果表明，相较于Minimal-hand与HOPE-Net模型，该模型预测关键点的mean_EPE分别降低0.882和0.710 mm，PSF@4分别提升31.38和32.31个百分点。最后，为探索产业级应用，结合PKPD-Net进行应用开发，通过时序关联实现操作轨迹的复原。

关键词: 虚拟现实, 精细操作, 深度学习, 关键点估计, 特征融合

Abstract:

Virtual Reality(VR) is widely used in various fields. However, the need for refined operations can not be met when using existing interaction methods. Using an interactive pen can achieve an accurate input of 3D space and enhance productivity efficiency. Thus, this study proposes a two-stage key point estimation algorithm of interactive pen based on a single RGB picture, PKPD-Net. In particular, the 2D key points are first estimated using the Convolutional Block Attention Module-Stacked Hourglass Network(CBAM-SHN). Then, the location information of 3D key points is calculated based on the 2D posture characteristics. This model proposes an improved fusion method based on the CBAM modules, sub-pixel positioning of key points based on Offset, and auxiliary estimation through the key points of supplementary hands. As a result, it achieves a highly precise estimation of the 3D key points. This provides accurate location information for refined operations through interactive pens. The model training and testing are performed on numerous datasets.The PKPD-Net achieves a mean End Point Error(mean EPE) of the key points that is lower by 0.882 and 0.710 mm, compared to that obtained using Minimal-hand and HOPE-Net models, respectively. Moreover, the Percentage of Success Frame with less than 4 mm(PSF@4) of key points achieved using the proposed model is higher by 31.38 and 32.31 percentage points, respectively. Thus, the method proposed in this study proves to be more advanced and effective than the existing methods. Finally, to explore the product applications, PKPD-Net is used to recover the operating trajectory through a time-sequential association.

Key words: Virtual Reality(VR), refined operation, deep learning, key point estimation, feature fusion

朱兴帅, 叶彬, 姚康, 丁上上, 徐道亮, 付威威. 基于深度学习的交互笔关键点估计研究[J]. 计算机工程, 2023, 49(10): 22-30.

Xingshuai ZHU, Bin YE, Kang YAO, Shangshang DING, Daoliang XU, Weiwei FU. Research on Key Point Estimation of Interactive Pen Based on Deep Learning[J]. Computer Engineering, 2023, 49(10): 22-30.

http://www.ecice06.com/CN/Y2023/V49/I10/22

图/表 14

图1 PKPD-Net网络结构

Fig.1 PKPD-Net network structure

图2 基于Offset偏移量的关键点亚像素定位

Fig.2 Sub-pixel positioning of key point based on Offset

图3 Hourglass Module网络结构

Fig.3 Hourglass Module network structure

图4 CBAM网络结构

Fig.4 CBAM network structure

图5 CBAM-SHN网络结构

Fig.5 CBAM-SHN network structure

图6 三维关键点估计网络结构

Fig.6 3D key point estimation network structure

图7 23个关键点标注规则

Fig.7 23 key points marking rules

图8 可视化估计结果对比

Fig.8 Comparison of visual estimation results

图9 书写实验示意图

Fig.9 Schematic diagrams of writing experiment

参考文献 27

1	杨青, 钟书华. 国外"虚拟现实技术发展及演化趋势"研究综述. 自然辩证法通讯, 2021, 43 (3): 97- 106. URL
	YANG Q , ZHONG S H . A review of foreign countries on the development and evolution trends of virtual reality technology. Journal of Dialectics of Nature, 2021, 43 (3): 97- 106. URL
2	KIRSH D . Thinking with external representations. AI & Society, 2010, 25 (4): 441- 454.
3	李柯泉, 陈燕, 刘佳晨, 等. 基于深度学习的目标检测算法综述. 计算机工程, 2022, 48 (7): 1- 12. URL
	LI K Q , CHEN Y , LIU J C , et al. Survey of deep learning-based object detection algorithms. Computer Engineering, 2022, 48 (7): 1- 12. URL
4	HE C H, LI R H, LI S, et al. Voxel set transformer: a set-to-set approach to 3D object detection from point clouds[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2022: 8407-8417.
5	YANG L H, ZHUO W, QI L, et al. ST: make self-training work better for semi-supervised semantic segmentation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2022: 4258-4267.
6	CHEN Z Z, WANG T, WU X W, et al. Class re-activation maps for weakly-supervised semantic segmentation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2022: 959-968.
7	李杰. 基于深度卷积神经网络的关键点检测算法研究. 成都: 中国科学院光电技术研究所, 2022, URL
	LI J . Research on keypoint detection algorithm based on deep convolutional neural network. Chengdu: Institute of Optics and Electronics, Chinese Academy of Sciences, 2022, URL
8	WANG Z T, NIE X C, QU X C, et al. Distribution-aware single-stage models for multi-person 3D pose estimation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2022: 13086-13095.
9	PAVLAKOS G, ZHOU X W, DERPANIS K G, et al. Coarse-to-fine volumetric prediction for single-image 3D human pose[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2017: 1263-1272.
10	SUN X, XIAO B, WEI F Y, et al. Integral human pose regression[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 536-553.
11	ZHAN Y, LI F H, WENG R L, et al. Ray3D: ray-based 3D human pose estimation for monocular absolute 3D localization[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2022: 13106-13115.
12	CHEN Y J, TU Z G, KANG D, et al. Model-based 3D hand reconstruction via self-supervised learning[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2021: 10446-10455.
13	LIU S W, JIANG H W, XU J R, et al. Semi-supervised 3D hand-object poses estimation with interactions in time[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2021: 14682-14692.
14	MARTINEZ J, HOSSAIN R, ROMERO J, et al. A simple yet effective baseline for 3d human pose estimation[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2017: 2659-2668.
15	LI C, LEE G H. Generating multiple hypotheses for 3D human pose estimation with mixture density network[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2020: 9879-9887.
16	CHEN C H, RAMANAN D. 3D human pose estimation=2D pose estimation matching[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2017: 5759-5767.
17	ZIMMERMANN C, BROX T. Learning to estimate 3D hand pose from single RGB images[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2017: 4913-4921.
18	MUELLER F, BERNARD F, SOTNYCHENKO O, et al. GANerated hands for real-time 3D hand tracking from monocular RGB[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2018: 49-59.
19	ZHOU K, HAN X G, JIANG N J, et al. HEMlets pose: learning part-centric heatmap triplets for accurate 3D human pose estimation[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2020: 2344-2353.
20	DOOSTI B, NAHA S, MIRBAGHERI M, et al. HOPE-Net: a graph-based model for hand-object pose estimation[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2020: 6607-6616.
21	NEWELL A, YANG K Y, DENG J. Stacked hourglass networks for human pose estimation[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 483-499.
22	SZEGEDY C, IOFFE S, VANHOUCKE V, et al. Inception-v4, Inception-ResNet and the impact of residual connections on learning[EB/OL]. [2022-10-02]. https://arxiv.org/abs/1602.07261.
23	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2016: 770-778.
24	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 3-19.
25	黄青丹, 何彬彬, 宋浩永, 等. 基于双目立体视觉的目标空间坐标计算及姿态估计. 华南师范大学学报(自然科学版), 2020, 52 (2): 9- 13. URL
	HUANG Q D , HE B B , SONG H Y , et al. Space coordinate calculation and attitude estimation based on binocular stereo vision. Journal of South China Normal University(Natural Science Edition), 2020, 52 (2): 9- 13. URL
26	KINGMA D P, BA J. Adam: a method for stochastic optimization[EB/OL]. [2022-10-02]. https://arxiv.org/abs/1412.6980.
27	ZHOU Y X, HABERMANN M, XU W P, et al. Monocular real-time hand shape and motion capture using multi-modal data[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2020: 5345-5354.

[1]	范文卓, 吴涛, 许俊平, 李庆庆, 张建林, 李美惠, 魏宇星. 基于多分辨率特征融合的任意尺度图像超分辨率重建[J]. 计算机工程, 2023, 49(9): 217-225.
[2]	李嘉新, 侯进, 盛博莹, 周宇航. 基于改进YOLOv5的遥感小目标检测网络[J]. 计算机工程, 2023, 49(9): 256-264.
[3]	池亚平, 岳梓岩, 林雨衡. 基于Transformer的SM4算法工作模式识别[J]. 计算机工程, 2023, 49(9): 109-117.
[4]	林中霖, 时金桥, 王美琪, 王学宾, 王雨燕. 基于应用行为划分的Android恶意应用检测技术[J]. 计算机工程, 2023, 49(9): 125-136.
[5]	韩璐, 霍纬纲, 张永会, 刘涛. 基于多尺度特征融合与双注意力机制的多元时间序列预测[J]. 计算机工程, 2023, 49(9): 99-108.
[6]	江雨燕, 陶承凤, 李平. 数据增强和自适应自步学习的深度子空间聚类算法[J]. 计算机工程, 2023, 49(8): 96-103, 110.
[7]	宋志娜, 李莎, 杨建明, 徐川. 基于特征与区域定位增强的遥感舰船目标检测[J]. 计算机工程, 2023, 49(8): 257-264.
[8]	张欣怡, 张飞, 郝斌, 高鹭, 任晓颖. 基于改进YOLOv5的口罩佩戴检测算法[J]. 计算机工程, 2023, 49(8): 265-274.
[9]	李泽水, 冀俊忠, 杨翠翠. 基于边权重信息深度网络嵌入的PPIN功能模块检测[J]. 计算机工程, 2023, 49(8): 69-76.
[10]	杨长沛, 廖列法. 基于门控空洞卷积特征融合的中文命名实体识别[J]. 计算机工程, 2023, 49(8): 85-95.
[11]	王可铮, 徐玉芬, 周尚波. 结合对比感知损失和融合注意力的图像去雾模型[J]. 计算机工程, 2023, 49(8): 207-214.
[12]	杨祖赫, 黎智辉, 唐云祁, 晏于文, 宋华青. 结合语义与图像信息的行人属性识别算法[J]. 计算机工程, 2023, 49(8): 215-222, 231.
[13]	刘俊豪, 王美林, 谢兴, 宋烨兴, 许莉花. 基于改进YOLOv5的皮革瑕疵检测算法[J]. 计算机工程, 2023, 49(8): 240-249.
[14]	陈露萌, 曹彦彦, 黄民, 谢鑫钢. 基于改进YOLOv5的火焰检测方法[J]. 计算机工程, 2023, 49(8): 291-301, 309.
[15]	闫兴亚, 匡娅茜, 白光睿, 李月. 基于深度学习的学生课堂行为识别方法[J]. 计算机工程, 2023, 49(7): 251-258.

选择文件类型/文献管理软件名称

选择包含的内容

基于深度学习的交互笔关键点估计研究

Research on Key Point Estimation of Interactive Pen Based on Deep Learning

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 27

相关文章 15

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于深度学习的交互笔关键点估计研究

Research on Key Point Estimation of Interactive Pen Based on Deep Learning

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 27

相关文章 15

编辑推荐

Metrics

本文评价