融合注意力机制的多视图卷积手势识别研究

doi:10.19678/j.issn.1000-3428.0067439

摘要/Abstract

摘要：

基于表面肌电信号（sEMG）的手势识别在人机交互中发挥着重要作用，然而，由于sEMG具有非线性和随机性，因此提升基于稀疏多通道sEMG的手势识别准确率难度较高。提出一种融合注意力机制的多视图卷积手势识别模型。首先使用200 ms滑动窗口提取经典的sEMG特征集构建多视图输入，其次利用高效通道注意力对多视图特征在通道维度进行加权，以强化有效特征同时弱化无效特征，最后通过多视图卷积从带有注意力权重的肌电特征中提取高层特征，利用高层特征融合模块对其进行融合以降低数据维度并提高模型鲁棒性。在NinaPro DB1、NinaPro DB5、NinaPro DB7 3个肌电公共数据集上进行训练和评估，结果表明，该模型在200 ms滑动采样窗口上的平均识别准确率分别为87.98%、94.97%和89.67%，整段手势动作的平均投票准确率分别为97.38%、98.41%和97.09%，平均信息传输率为1 308.71 bit/min。与传统机器学习方法和近年来前沿的深度学习手势识别方法相比，所提模型在单模态肌电和多模态手势识别上均具有更高的识别准确率，验证了其有效性和通用性。

关键词: 表面肌电信号, 手势识别, 特征提取, 注意力机制, 多视图卷积

Abstract:

Gesture recognition based on surface Electromyography(sEMG) plays an important role in human-computer interactions. However, improving the accuracy of gesture recognition is a challenging task because of the nonlinearity and randomness of sEMG. To this end, this paper proposes a multiview convolutional gesture recognition model that incorporates an attention mechanism. First, a multiview input is constructed by extracting the classical feature set of the sEMG signal using a 200 ms sliding window. Second, Efficient Channel Attention(ECA) is used to weight the multiview features in the channel dimension, to strengthen effective features and weaken ineffective ones. Finally, multiview convolution is used to extract the high-dimensional myoelectric features with attention weights, thereby fusing them using the high-level feature fusion module to reduce data dimensionality and improve model robustness. The models were trained and evaluated on three public EMG datasets, namely NinaPro DB1, NinaPro DB5 and NinaPro DB7, obtaining an average recognition accuracy of 87.98%, 94.97%, 89.67%, respectively over a 200 ms sliding sampling window; the average voting accuracy for the entire gesture movement was 97.38%, 98.41%, 97.09%, respectively, and the average information transfer rate was 1 308.71 bit/min. Compared with traditional machine learning methods and state-of-the-art deep gesture recognition methods that have been developed in recent years, the present model has higher recognition accuracy for both unimodal myoelectric and multi-modal gesture recognition, proving its effectiveness and generality.

Key words: surface Electromyography(sEMG), gesture recognition, feature extraction, attention mechanism, multiview convolution

袁文涛, 卫文韬, 高德民. 融合注意力机制的多视图卷积手势识别研究[J]. 计算机工程, 2024, 50(3): 208-215.

Wentao YUAN, Wentao WEI, Demin GAO. Research on Multiview Convolutional Gesture Recognition with Fusion Attention Mechanism[J]. Computer Engineering, 2024, 50(3): 208-215.

https://www.ecice06.com/CN/Y2024/V50/I3/208

图/表 12

图1 原始sEMG滤波前后对比

Fig.1 Comparison of the original sEMG before and after filtering

图2 通道重排列算法示例

Fig.2 Example of channel rearrangement algorithm

图3 多视图卷积注意力网络框架

Fig.3 Multiview convolutional attention network framework

图4 ECA注意力模块结构

Fig.4 ECA attention module structure

图5 示例手势的sEMG对比

Fig.5 sEMG comparison of example gestures

图6 NinaPro DB5数据集中每个被试者的消融实验结果

Fig.6 Ablation experiment results of each subject in the NinaPro DB5 dataset

参考文献 28

1	MEDVED V, MEDVED S, KOVAC I. Critical appraisal of surface Electromyography(sEMG) as a taught subject and clinical tool in medicine and kinesiology. Frontiers in Neurology, 2020, 11, 560363. doi: 10.3389/fneur.2020.560363
2	SUN Y, XU C, LI G F, et al. Intelligent human computer interaction based on non redundant EMG signal. Alexandria Engineering Journal, 2020, 59(3): 1149- 1157. doi: 10.1016/j.aej.2020.01.015
3	PALERMO F, COGNOLATO M, GIJSBERTS A, et al. Repeatability of grasp recognition for robotic hand prosthesis control based on sEMG data[C]//Proceedings of International Conference on Rehabilitation Robotics. Washington D. C., USA: IEEE Press, 2017: 1154-1159.
4	周旭峰, 王醒策, 武仲科, 等. 基于组合RNN网络的EMG信号手势识别. 光学精密工程, 2020, 28(2): 424- 442. URL
	ZHOU X F, WANG X C, WU Z K, et al. Gesture recognition with EMG signals based on ensemble RNN. Optics and Precision Engineering, 2020, 28(2): 424- 442. URL
5	谷学静, 沈攀, 刘海望, 等. 表面肌电信号的多流卷积网络融合手势识别方法. 计算机应用与软件, 2022, 39(8): 220- 225. doi: 10.3969/j.issn.1000-386x.2022.08.032
	GU X J, SHEN P, LIU H W, et al. Gesture recognition based on multi-stream convolution network for surface emg signal. Computer Applications and Software, 2022, 39(8): 220- 225. doi: 10.3969/j.issn.1000-386x.2022.08.032
6	LI W, SHI P, YU H L. Gesture recognition using surface electromyography and deep learning for prostheses hand: state-of-the-art, challenges, and future. Frontiers in Neuroscience, 2021, 15, 621885. doi: 10.3389/fnins.2021.621885
7	HOUSTON M, WU A, ZHANG Y C. Optimizing input for gesture recognition using convolutional networks on HD-sEMG instantaneous images[C]//Proceedings of the 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society. Washington D. C., USA: IEEE Press, 2021: 6539-6542.
8	范秀琴, 喻洪流, 李素姣. 基于BiLSTM的表面肌电图手势识别算法. 生物医学工程学进展, 2021, 42(2): 92- 96. doi: 10.3969/j.issn.1674-1242.2021.02.007
	FAN X Q, YU H L, LI S J. Gesture recognition of sEMG based on BiLSTM. Progress in Biomedical Engineering, 2021, 42(2): 92- 96. doi: 10.3969/j.issn.1674-1242.2021.02.007
9	AROZI M, CAESARENDRA W, ARIYANTO M, et al. Pattern recognition of single-channel sEMG signal using PCA and ANN method to classify nine hand movements. Symmetry, 2020, 12(4): 541. doi: 10.3390/sym12040541
10	MENDES J J A JR, FREITAS M L B, SIQUEIRA H V, et al. Feature selection and dimensionality reduction: an extensive comparison in hand gesture classification by sEMG in eight channels armband approach. Biomedical Signal Processing and Control, 2020, 59, 101920. doi: 10.1016/j.bspc.2020.101920
11	ATZORI M, GIJSBERTS A, CASTELLINI C, et al. Electromyography data for non-invasive naturally-controlled robotic hand prostheses. Scientific Data, 2014, 1, 140053. doi: 10.1038/sdata.2014.53
12	KRASOULIS A, KYRANOU I, ERDEN M S, et al. Improved prosthetic hand control with concurrent use of myoelectric and inertial measurements. Journal of Neuroengineering and Rehabilitation, 2017, 14(1): 71. doi: 10.1186/s12984-017-0284-4
13	GENG W D, DU Y, JIN W G, et al. Gesture recognition by instantaneous surface EMG images. Scientific Reports, 2016, 6, 36571. doi: 10.1038/srep36571
14	李沿宏, 江茜, 邹可, 等. 融合注意力机制的多流卷积肌电手势识别网络. 计算机应用研究, 2021, 38(11): 3258- 3263. URL
	LI Y H, JIANG X, ZOU K, et al. Multi-stream convolutional myoelectric gesture recognition networks fusing attentional mechanisms. Application Research of Computers, 2021, 38(11): 3258- 3263. URL
15	卫文韬, 李亚军. 基于双流卷积神经网络的肌电信号手势识别方法. 计算机集成制造系统, 2022, 28(1): 124- 131. URL
	WEI W T, LI Y J. Surface electromyography based gesture recognition based on dual-stream CNN. Computer Integrated Manufacturing Systems, 2022, 28(1): 124- 131. URL
16	HU Y, WONG Y, WEI W T, et al. A novel attention-based hybrid CNN-RNN architecture for sEMG-based gesture recognition. PLoS One, 2018, 13(10): e0206049. doi: 10.1371/journal.pone.0206049
17	WEI W T, DAI Q F, WONG Y, et al. Surface-electromyography-based gesture recognition by multi-view deep learning. IEEE Transactions on Bio-Medical Engineering, 2019, 66(10): 2964- 2973. doi: 10.1109/TBME.2019.2899222
18	PHINYOMARK A, PHUKPATTARANONT P, LIMSAKUL C. Feature reduction and selection for EMG signal classification. Expert Systems with Applications, 2012, 39(8): 7420- 7431. doi: 10.1016/j.eswa.2012.01.102
19	JIANG W C, YIN Z Z. Human activity recognition using wearable sensors by deep convolutional neural networks[C]//Proceedings of the 23rd ACM International Conference on Multimedia. New York, USA: ACM Press, 2015: 1307-1310.
20	POWAR O S, CHEMMANGAT K. Comparison of pre-processing filters on the performance of sEMG based pattern recognition[C]//Proceedings of International Conference on Power Electronics Applications and Technology in Present Energy Scenario. Washington D. C., USA: IEEE Press, 2019: 1-6.
21	SANDOVAL-ESPINO J A, ZAMUDIO-LARA A, MARBÁN-SALGADO J A, et al. Selection of the best set of features for sEMG-based hand gesture recognition applying a CNN architecture. Sensors, 2022, 22(13): 4972. doi: 10.3390/s22134972
22	李翔, 张涛, 张哲, 等. Transformer在计算机视觉领域的研究综述. 计算机工程与应用, 2023, 59(1): 1- 14. URL
	LI X, ZHANG T, ZHANG Z, et al. Survey of Transformer research in computer vision. Computer Engineering and Applications, 2023, 59(1): 1- 14. URL
23	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7132-7141.
24	GUO W Z, WANG J W, WANG S P. Deep multimodal representation learning: a survey. IEEE Access, 2019, 7, 63373- 63394. doi: 10.1109/ACCESS.2019.2916887
25	HAN X, ZHANG Z Y, DING N, et al. Pre-trained models: past, present and future. AI Open, 2021, 2, 225- 250. doi: 10.1016/j.aiopen.2021.08.002
26	WANG Y J, GAO X R, HONG B, et al. Brain-computer interfaces based on visual evoked potentials. IEEE Engineering in Medicine and Biology Magazine, 2008, 27(5): 64- 71. doi: 10.1109/MEMB.2008.923958
27	PIZZOLATO S, TAGLIAPIETRA L, COGNOLATO M, et al. Comparison of six electromyography acquisition setups on hand movement classification tasks. PLoS One, 2017, 12(10): e0186132. doi: 10.1371/journal.pone.0186132
28	SRI-IESARANUSORN P, CHAIYAROJ A, BUEKBAN C, et al. Classification of 41 hand and wrist movements via surface electromyogram using deep neural network. Frontiers in Bioengineering and Biotechnology, 2021, 9, 548357. doi: 10.3389/fbioe.2021.548357

[1]	王炼红, 林飞鹏, 李潇瑶, 谌桂枝, 周莉. 融入课程知识图谱的KMAKT预测[J]. 计算机工程, 2024, 50(7): 23-31.
[2]	钱来, 赵卫伟. 基于对比学习和注意力机制的文本分类方法[J]. 计算机工程, 2024, 50(7): 104-111.
[3]	刘建敏, 林晖, 汪晓丁. 基于图注意力机制的无地图场景轨迹预测方法[J]. 计算机工程, 2024, 50(7): 144-153.
[4]	屠乃威, 焦猛, 阎馨. 复杂环境下输电线路鸟巢目标图像检测模型[J]. 计算机工程, 2024, 50(7): 216-226.
[5]	肖慈, 徐杨, 张永丹, 冯明文, 黄易仟. 结合注意力和低光增强的夜间语义分割[J]. 计算机工程, 2024, 50(7): 271-281.
[6]	张锡英, 孙守东, 于海浩, 边继龙. 基于空间传播的多视图三维重建[J]. 计算机工程, 2024, 50(7): 293-302.
[7]	贵向泉, 刘世清, 李立, 秦庆松, 李唐艳. 基于改进YOLOv8的景区行人检测算法[J]. 计算机工程, 2024, 50(7): 342-351.
[8]	程腾腾, 姚春龙, 于晓强, 李旭, 王庆丰. 基于多头注意力机制融合常识知识的共情对话生成[J]. 计算机工程, 2024, 50(6): 94-101.
[9]	胡庆. 多尺度融合与双输出U-Net网络的行人重识别[J]. 计算机工程, 2024, 50(6): 102-109.
[10]	更藏措毛, 黄鹤鸣, 杨毅杰. 融合多尺度特征与上下文信息的语音增强方法[J]. 计算机工程, 2024, 50(6): 138-147.
[11]	贺姗, 蔺素珍, 王彦博, 李大威. 基于特征融合的多波段图像描述生成方法[J]. 计算机工程, 2024, 50(6): 236-244.
[12]	杨硕, 王一丁. 基于改进薄板样条运动模型的人脸动画算法[J]. 计算机工程, 2024, 50(6): 255-265.
[13]	陈晓玉, 沈晨, 沈阅, 孔德明. 基于改进SwiftNet的堆场图像实时分割网络[J]. 计算机工程, 2024, 50(6): 296-303.
[14]	游奔, 李晓红, 姚锦, 冯绍杰. 基于多粒度图与注意力机制的半监督短文本分类[J]. 计算机工程, 2024, 50(5): 83-90.
[15]	代巍, 王丰羽, 冀常鹏. 基于情感增强与双图卷积网络的方面级情感分析[J]. 计算机工程, 2024, 50(5): 120-127.

选择文件类型/文献管理软件名称

选择包含的内容