Research on Multiview Convolutional Gesture Recognition with Fusion Attention Mechanism

doi:10.19678/j.issn.1000-3428.0067439

Abstract

Abstract:

Gesture recognition based on surface Electromyography(sEMG) plays an important role in human-computer interactions. However, improving the accuracy of gesture recognition is a challenging task because of the nonlinearity and randomness of sEMG. To this end, this paper proposes a multiview convolutional gesture recognition model that incorporates an attention mechanism. First, a multiview input is constructed by extracting the classical feature set of the sEMG signal using a 200 ms sliding window. Second, Efficient Channel Attention(ECA) is used to weight the multiview features in the channel dimension, to strengthen effective features and weaken ineffective ones. Finally, multiview convolution is used to extract the high-dimensional myoelectric features with attention weights, thereby fusing them using the high-level feature fusion module to reduce data dimensionality and improve model robustness. The models were trained and evaluated on three public EMG datasets, namely NinaPro DB1, NinaPro DB5 and NinaPro DB7, obtaining an average recognition accuracy of 87.98%, 94.97%, 89.67%, respectively over a 200 ms sliding sampling window; the average voting accuracy for the entire gesture movement was 97.38%, 98.41%, 97.09%, respectively, and the average information transfer rate was 1 308.71 bit/min. Compared with traditional machine learning methods and state-of-the-art deep gesture recognition methods that have been developed in recent years, the present model has higher recognition accuracy for both unimodal myoelectric and multi-modal gesture recognition, proving its effectiveness and generality.

Key words: surface Electromyography(sEMG), gesture recognition, feature extraction, attention mechanism, multiview convolution

摘要：

基于表面肌电信号（sEMG）的手势识别在人机交互中发挥着重要作用，然而，由于sEMG具有非线性和随机性，因此提升基于稀疏多通道sEMG的手势识别准确率难度较高。提出一种融合注意力机制的多视图卷积手势识别模型。首先使用200 ms滑动窗口提取经典的sEMG特征集构建多视图输入，其次利用高效通道注意力对多视图特征在通道维度进行加权，以强化有效特征同时弱化无效特征，最后通过多视图卷积从带有注意力权重的肌电特征中提取高层特征，利用高层特征融合模块对其进行融合以降低数据维度并提高模型鲁棒性。在NinaPro DB1、NinaPro DB5、NinaPro DB7 3个肌电公共数据集上进行训练和评估，结果表明，该模型在200 ms滑动采样窗口上的平均识别准确率分别为87.98%、94.97%和89.67%，整段手势动作的平均投票准确率分别为97.38%、98.41%和97.09%，平均信息传输率为1 308.71 bit/min。与传统机器学习方法和近年来前沿的深度学习手势识别方法相比，所提模型在单模态肌电和多模态手势识别上均具有更高的识别准确率，验证了其有效性和通用性。

关键词: 表面肌电信号, 手势识别, 特征提取, 注意力机制, 多视图卷积

Wentao YUAN, Wentao WEI, Demin GAO. Research on Multiview Convolutional Gesture Recognition with Fusion Attention Mechanism[J]. Computer Engineering, 2024, 50(3): 208-215.

袁文涛, 卫文韬, 高德民. 融合注意力机制的多视图卷积手势识别研究[J]. 计算机工程, 2024, 50(3): 208-215.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0067439

http://www.ecice06.com/EN/Y2024/V50/I3/208

Figures/Tables 12

Fig.1 Comparison of the original sEMG before and after filtering

Fig.2 Example of channel rearrangement algorithm

Fig.3 Multiview convolutional attention network framework

Fig.4 ECA attention module structure

Fig.5 sEMG comparison of example gestures

Fig.6 Ablation experiment results of each subject in the NinaPro DB5 dataset

References 28

1	MEDVED V, MEDVED S, KOVAC I. Critical appraisal of surface Electromyography(sEMG) as a taught subject and clinical tool in medicine and kinesiology. Frontiers in Neurology, 2020, 11, 560363. doi: 10.3389/fneur.2020.560363
2	SUN Y, XU C, LI G F, et al. Intelligent human computer interaction based on non redundant EMG signal. Alexandria Engineering Journal, 2020, 59(3): 1149- 1157. doi: 10.1016/j.aej.2020.01.015
3	PALERMO F, COGNOLATO M, GIJSBERTS A, et al. Repeatability of grasp recognition for robotic hand prosthesis control based on sEMG data[C]//Proceedings of International Conference on Rehabilitation Robotics. Washington D. C., USA: IEEE Press, 2017: 1154-1159.
4	周旭峰, 王醒策, 武仲科, 等. 基于组合RNN网络的EMG信号手势识别. 光学精密工程, 2020, 28(2): 424- 442. URL
	ZHOU X F, WANG X C, WU Z K, et al. Gesture recognition with EMG signals based on ensemble RNN. Optics and Precision Engineering, 2020, 28(2): 424- 442. URL
5	谷学静, 沈攀, 刘海望, 等. 表面肌电信号的多流卷积网络融合手势识别方法. 计算机应用与软件, 2022, 39(8): 220- 225. doi: 10.3969/j.issn.1000-386x.2022.08.032
	GU X J, SHEN P, LIU H W, et al. Gesture recognition based on multi-stream convolution network for surface emg signal. Computer Applications and Software, 2022, 39(8): 220- 225. doi: 10.3969/j.issn.1000-386x.2022.08.032
6	LI W, SHI P, YU H L. Gesture recognition using surface electromyography and deep learning for prostheses hand: state-of-the-art, challenges, and future. Frontiers in Neuroscience, 2021, 15, 621885. doi: 10.3389/fnins.2021.621885
7	HOUSTON M, WU A, ZHANG Y C. Optimizing input for gesture recognition using convolutional networks on HD-sEMG instantaneous images[C]//Proceedings of the 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society. Washington D. C., USA: IEEE Press, 2021: 6539-6542.
8	范秀琴, 喻洪流, 李素姣. 基于BiLSTM的表面肌电图手势识别算法. 生物医学工程学进展, 2021, 42(2): 92- 96. doi: 10.3969/j.issn.1674-1242.2021.02.007
	FAN X Q, YU H L, LI S J. Gesture recognition of sEMG based on BiLSTM. Progress in Biomedical Engineering, 2021, 42(2): 92- 96. doi: 10.3969/j.issn.1674-1242.2021.02.007
9	AROZI M, CAESARENDRA W, ARIYANTO M, et al. Pattern recognition of single-channel sEMG signal using PCA and ANN method to classify nine hand movements. Symmetry, 2020, 12(4): 541. doi: 10.3390/sym12040541
10	MENDES J J A JR, FREITAS M L B, SIQUEIRA H V, et al. Feature selection and dimensionality reduction: an extensive comparison in hand gesture classification by sEMG in eight channels armband approach. Biomedical Signal Processing and Control, 2020, 59, 101920. doi: 10.1016/j.bspc.2020.101920
11	ATZORI M, GIJSBERTS A, CASTELLINI C, et al. Electromyography data for non-invasive naturally-controlled robotic hand prostheses. Scientific Data, 2014, 1, 140053. doi: 10.1038/sdata.2014.53
12	KRASOULIS A, KYRANOU I, ERDEN M S, et al. Improved prosthetic hand control with concurrent use of myoelectric and inertial measurements. Journal of Neuroengineering and Rehabilitation, 2017, 14(1): 71. doi: 10.1186/s12984-017-0284-4
13	GENG W D, DU Y, JIN W G, et al. Gesture recognition by instantaneous surface EMG images. Scientific Reports, 2016, 6, 36571. doi: 10.1038/srep36571
14	李沿宏, 江茜, 邹可, 等. 融合注意力机制的多流卷积肌电手势识别网络. 计算机应用研究, 2021, 38(11): 3258- 3263. URL
	LI Y H, JIANG X, ZOU K, et al. Multi-stream convolutional myoelectric gesture recognition networks fusing attentional mechanisms. Application Research of Computers, 2021, 38(11): 3258- 3263. URL
15	卫文韬, 李亚军. 基于双流卷积神经网络的肌电信号手势识别方法. 计算机集成制造系统, 2022, 28(1): 124- 131. URL
	WEI W T, LI Y J. Surface electromyography based gesture recognition based on dual-stream CNN. Computer Integrated Manufacturing Systems, 2022, 28(1): 124- 131. URL
16	HU Y, WONG Y, WEI W T, et al. A novel attention-based hybrid CNN-RNN architecture for sEMG-based gesture recognition. PLoS One, 2018, 13(10): e0206049. doi: 10.1371/journal.pone.0206049
17	WEI W T, DAI Q F, WONG Y, et al. Surface-electromyography-based gesture recognition by multi-view deep learning. IEEE Transactions on Bio-Medical Engineering, 2019, 66(10): 2964- 2973. doi: 10.1109/TBME.2019.2899222
18	PHINYOMARK A, PHUKPATTARANONT P, LIMSAKUL C. Feature reduction and selection for EMG signal classification. Expert Systems with Applications, 2012, 39(8): 7420- 7431. doi: 10.1016/j.eswa.2012.01.102
19	JIANG W C, YIN Z Z. Human activity recognition using wearable sensors by deep convolutional neural networks[C]//Proceedings of the 23rd ACM International Conference on Multimedia. New York, USA: ACM Press, 2015: 1307-1310.
20	POWAR O S, CHEMMANGAT K. Comparison of pre-processing filters on the performance of sEMG based pattern recognition[C]//Proceedings of International Conference on Power Electronics Applications and Technology in Present Energy Scenario. Washington D. C., USA: IEEE Press, 2019: 1-6.
21	SANDOVAL-ESPINO J A, ZAMUDIO-LARA A, MARBÁN-SALGADO J A, et al. Selection of the best set of features for sEMG-based hand gesture recognition applying a CNN architecture. Sensors, 2022, 22(13): 4972. doi: 10.3390/s22134972
22	李翔, 张涛, 张哲, 等. Transformer在计算机视觉领域的研究综述. 计算机工程与应用, 2023, 59(1): 1- 14. URL
	LI X, ZHANG T, ZHANG Z, et al. Survey of Transformer research in computer vision. Computer Engineering and Applications, 2023, 59(1): 1- 14. URL
23	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 7132-7141.
24	GUO W Z, WANG J W, WANG S P. Deep multimodal representation learning: a survey. IEEE Access, 2019, 7, 63373- 63394. doi: 10.1109/ACCESS.2019.2916887
25	HAN X, ZHANG Z Y, DING N, et al. Pre-trained models: past, present and future. AI Open, 2021, 2, 225- 250. doi: 10.1016/j.aiopen.2021.08.002
26	WANG Y J, GAO X R, HONG B, et al. Brain-computer interfaces based on visual evoked potentials. IEEE Engineering in Medicine and Biology Magazine, 2008, 27(5): 64- 71. doi: 10.1109/MEMB.2008.923958
27	PIZZOLATO S, TAGLIAPIETRA L, COGNOLATO M, et al. Comparison of six electromyography acquisition setups on hand movement classification tasks. PLoS One, 2017, 12(10): e0186132. doi: 10.1371/journal.pone.0186132
28	SRI-IESARANUSORN P, CHAIYAROJ A, BUEKBAN C, et al. Classification of 41 hand and wrist movements via surface electromyogram using deep neural network. Frontiers in Bioengineering and Biotechnology, 2021, 9, 548357. doi: 10.3389/fbioe.2021.548357

[1]	Bochao ZHAO, Jiajun MA, Lei CUI, Wenpeng LUAN, Jing ZHU. Anomaly Detection for Photovoltaic Based on Improved VMD-XGBoost-BiLSTM Combination Model [J]. Computer Engineering, 2024, 50(3): 306-316.
[2]	Jiayuan ZHAO, Yuru ZHANG, Xiaodong SU, Hongyan XU, Shizhou LI, Yurong ZHANG. Implicit Modeling Network of Human Keypoints Based on Attention Mechanism [J]. Computer Engineering, 2024, 50(3): 317-325.
[3]	Xinlin XIE, Dongxu YIN, Taoyuan ZHANG, Gang XIE. Multiscale Fusion Crowd Counting Algorithm Based on Attention Mechanism [J]. Computer Engineering, 2024, 50(3): 290-297.
[4]	Fangxin XU, Rong FAN, Xiaolu MA. Improved YOLOv7 Algorithm for Crowded Pedestrian Detection [J]. Computer Engineering, 2024, 50(3): 250-258.
[5]	Xiangzhen GUO, Sitong LI, Rui LU, Sen GUO, Xuerong CUI, Gang YANG. Colon Polyp Segmentation Network Based on Multi-task Joint Attention [J]. Computer Engineering, 2024, 50(2): 327-336.
[6]	Guohui DING, Yuqi LIU, Yankai WANG, Shizhan GENG, Tianhao JIANG. Research on Low-Correlation Sequence Data Prediction Based on Flip Network [J]. Computer Engineering, 2024, 50(2): 78-90.
[7]	Haochen XU, Manhua LIU. Facial Landmark Detection Based on Hierarchical Self-Attention Network [J]. Computer Engineering, 2024, 50(2): 239-246.
[8]	Shuaiwei LIU, Zhi LI, Guomei WANG, Li ZHANG. Adversarial Example Generation Algorithm Based on Transformer and GAN [J]. Computer Engineering, 2024, 50(2): 180-187.
[9]	Zhengjia WANG, Feifei HU, Chengjuan ZHANG, Zhuo LEI, Tao HE. Adaptive Window Stereo Matching Algorithm with Lightweight Transformer [J]. Computer Engineering, 2024, 50(2): 256-265.
[10]	Ruijun YANG, Jinjing QIN, Yan CHENG. Low-Light Enhancement Model in Natural Scenes Based on Generative Adversarial Network [J]. Computer Engineering, 2024, 50(1): 279-288.
[11]	Xiaofeng XU, Yunzhi HUANG, Jun XU. Dual-Branch Vascular Segmentation Model Based on Anisotropic Attention [J]. Computer Engineering, 2024, 50(1): 348-356.
[12]	Wenshun SHENG, Xiongfeng YU, Jiayan LIN, Xin CHEN. Small-Scale Object Detection Algorithm Integrating Attention and Feature Pyramids [J]. Computer Engineering, 2024, 50(1): 242-250.
[13]	Xiuyu SHEN, Weifeng JI, Yingqi LI, Xuan WU. TCA1C DDoS Detection Model for Edge Computing [J]. Computer Engineering, 2024, 50(1): 198-205.
[14]	Zhiqiang WU, Qing XIE, Lin LI, Yongjian LIU. Graph Neural Network Recommendation Algorithm Based on Multimodal Fusion [J]. Computer Engineering, 2024, 50(1): 91-100.
[15]	Lu HAN, Weigang HUO, Yonghui ZHANG, Tao LIU. Multivariate Time Series Forecasting Based on Multi-Scale Feature Fusion and Dual-Attention Mechanism [J]. Computer Engineering, 2023, 49(9): 99-108.

Please choose a citation manager

Content to export