Micro-Gesture Recognition Method Based on Multi-modal Collaborative Enhancement

doi:10.19678/j.issn.1000-3428.0252355

Abstract

Abstract: Micro-gestures are unconscious, subtle movements driven by intrinsic emotions. As they are capable of reflecting individuals' hidden emotional states, they hold significant value in affective computing. Micro-gestures exhibit transient characteristics in the temporal dimension and spatially ambiguous boundaries. Hence, these typical fine-grained behaviors pose substantial challenges for feature extraction by traditional methods. To address these issues, this paper proposes a micro-gesture recognition method based on multi-modal collaborative enhancement that constructs video, skeleton, and text modalities as a complementary representational triplet. This framework addresses the limitations of conventional vision-language models by introducing skeleton modality as a kinematic prior. By integrating visual context and semantic guidance, it establishes a multi-source complementary feature representation system. A dual-level collaborative module, called the Video—Pose Collaborative Module (VPCM), fuses fine-grained video details with global motion information from skeletons, and a cross-temporal attention mechanism is expands feature representations and enhances temporal modeling capability. The Text-Pose Collaborative Module (TPCM) introduces semantic priors from the text modality and adopts a Top-k-based fusion strategy to reinforce the semantic relevance of skeleton features, thereby improving the capture of fine-grained characteristics. To further optimize the multi-modal fusion performance, a two-stage training strategy is designed, in which unimodal encoders are pre-trained first, followed by collaborative learning through lightweight adapters and collaborative modules, thus effectively boosting model accuracy. Results of experiments on mainstream micro-gesture datasets demonstrate that the proposed model achieves a micro-gesture recognition accuracy of 70.40%, outperforming the current best-performing methods.

Key words: action recognition, multi-modal, CLIP model, micro-gesture, fine-grained

摘要： 微姿态是由内在情感驱动的无意识细微动作，能够反映个体隐藏情绪，在情感计算中具有重要价值。其在时间维度上具有瞬时性，在空间维度上幅度微小、边界模糊，属于典型的细粒度行为，传统方法难以提取有效特征。为此，提出一种基于多模态协同增强的微姿态识别方法，将视频、骨架与文本构建为互补表征三元组。该框架突破传统视觉-语言模型的局限性，引入骨架模态作为运动学先验，结合视觉上下文和语义引导，构建多源互补的特征表征体系。此外，提出双层级协同模块：视频-姿态协同模块（VPCM），融合视频的细节特征与骨架的全局运动信息，采用跨时间注意力机制扩展特征表示，增强时序建模能力；文本-姿态协同模块（TPCM），引入文本模态的语义先验，采用基于Top-k的融合策略强化骨架特征的语义关联性，提升对细粒度特征的捕获效果。为进一步优化多模态融合性能，提出两阶段训练策略，先对单模态编码器进行预训练，再通过轻量化适配器与协同模块进行协同学习，有效提升了模型的精度。在主流微姿态数据集上的实验结果表明，所提模型的识别准确率超越了当前最优方法，达到了70.40%的精度。

关键词: 动作识别, 多模态, CLIP模型, 微姿态, 细粒度

CLC Number:

TP391

LIU Yujie, WANG Yiwen. Micro-Gesture Recognition Method Based on Multi-modal Collaborative Enhancement[J]. Computer Engineering, 2026, 52(11): 319-330.

刘玉杰, 王一雯. 基于多模态协同增强的微姿态识别方法[J]. 计算机工程, 2026, 52(11): 319-330.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0252355

https://www.ecice06.com/EN/Y2026/V52/I11/319

References

[1] LIU X, SHI H L, CHEN H Y, et al. iMiGUE: an identity-free video dataset for micro-gesture understanding and emotion analysis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2021: 10631-10642.
[2] AVIEZER H, TROPE Y, TODOROV A. Body cues, not facial expressions, discriminate between intense positive and negative emotions[J]. Science, 2012, 338(6111): 1225-1229.
[3] AXTELL R E. Gestures: the do’s and taboos of body language around the world[M]. New Jersey, USA:John Wiley & Sons, Inc., 1991.
[4] 王园园, 曹慧,王廷蔚.基于多粒度交叉注意力的骨架动作识别方法[J/OL].计算机工程:1-10[2025-03-15].https://doi.org/10.19678/j.issn.1000-3428.0252088.WANG Y Y, CAO H, WANG T W. Skeleton-based action recognition method based on multi-granularity cross attention [J/OL]. Computer Engineering: 1-10 [2025-03-15].https://doi.org/10.19678/j.issn.1000-3428.0252088. (in Chinese)
[5] YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Washington D.C., USA: AAAI Press, 2018: 7444-7452.
[6] LIU Z Y, ZHANG H W, CHEN Z H, et al. Disentangling and unifying graph convolutions for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2020: 143-152.
[7] DAS S, SHARMA S, DAI R, et al. VPN: learning video-pose embedding for activities of daily living[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 72-90.
[8] AHN D, KIM S, HONG H, et al. STAR-Transformer: a spatio-temporal cross attention transformer for human action recognition[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Washington D.C., USA: IEEE Press, 2023: 3319-3328.
[9] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the International Conference on Machine Learning.New York, USA: ACM Press, 2021: 8748-8763.
[10] WANG M M, XING J Z, LIU Y. ActionCLIP: a new paradigm for video action recognition[EB/OL].[2025-03-15]. https://arxiv.org/abs/2109.08472.
[11] HUANG X H, ZHOU H, YAO K, et al. FROSTER: frozen CLIP is a strong teacher for open-vocabulary action recognition[EB/OL].[2025-03-15]. https://arxiv.org/abs/2402.03241.
[12] DUAN H D, ZHAO Y, CHEN K, et al. Revisiting skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE Press, 2022: 2969-2978.
[13] WANG Y, DONG Z, LI P, et al. A multimodal micro-gesture classification model based on CLIP[C]//Proceedings of the IJCAI 2024 Workshop & Challenge on Micro-Gesture Analysis for Hidden Emotion Understanding. Jeju, Republic of Korea:[s.n], 2024:1-9.
[14] SIMONYAN K, ZISSERMAN A. Two-stream convolutional networks for action recognition in videos[C]//Proceedings of the Advances in Neural Information Processing Systems. New York, USA: Curran Associates, Inc., 2014: 568-576.
[15] KARPATHY A, TODERICI G, SHETTY S, et al. Large-scale video classification with convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE Press, 2014: 1725-1732.
[16] 张聪聪, 何宁, 孙琪翔, 等. 基于注意力机制的3D DenseNet人体动作识别方法[J]. 计算机工程, 2021, 47(11): 313-320.ZHANG C C, HE N, SUN Q X, et al. Human motion recognition method based on attention mechanism of 3D DenseNet[J]. Computer Engineering, 2021, 47(11): 313-320. (in Chinese)
[17] CARREIRA J, ZISSERMAN A. Quo vadis, action recognition? a new model and the kinetics dataset[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE Press, 2017: 4724-4733.
[18] SOOMRO K, ZAMIR A R, SHAH M. UCF101: a dataset of 101 human actions classes from videos in the wild[EB/OL].[2025-03-15]. https://arxiv.org/abs/1212.0402.
[19] CHENG Q, CHENG J, LIU Z, et al. A dense-sparse complementary network for human action recognition based on RGB and skeleton modalities[J]. Expert Systems with Applications, 2024, 244: 123061.
[20] KIM S, AHN D, KO B C. Cross-modal learning with 3D deformable attention for action recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Paris, France: IEEE Press, 2023: 10231-10241.
[21] SIGURDSSON G A, GUPTA A, SCHMID C, et al. Charades-Ego: a large-scale dataset of paired third and first person videos[EB/OL].[2025-03-15]. https://arxiv.org/abs/1804.09626.
[22] PAN J T, LIN Z Y, ZHU X T, et al. ST-Adapter: parameter-efficient image-to-video transfer learning[C]//Proceedings of the Advances in Neural Information Processing Systems 35. New Orleans, USA: Neural Information Processing Systems Foundation, Inc., 2022: 26462-26477.
[23] CAO K D, WEI C, GAIDON A, et al. Learning imbalanced datasets with label-distribution-aware margin loss[C]//Proceedings of the Advances in Neural Information Processing Systems. New York, USA: Curran Associates, Inc., 2019: 1567-1578.
[24] KUEHNE H, JHUANG H, GARROTE E, et al. HMDB: a large video database for human motion recognition[C]//Proceedings of the International Conference on Computer Vision. Barcelona, Spain: IEEE Press, 2011: 2556-2563.
[25] ZHOU Y X, CHENG Z Q, LI C, et al. Hypergraph transformer for skeleton-based action recognition[EB/OL].[2025-03-15]. https://arxiv.org/abs/2211.09590.
[26] HUANG H X, GUO X P, PENG W, et al. Micro-gesture classification based on ensemble hypergraph-convolution Transformer[EB/OL].[2025-03-15]. https://ceur-ws.org/Vol-3522/paper_1.pdf.
[27] ZHOU B L, ANDONIAN A, OLIVA A, et al. Temporal relational reasoning in videos[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 831-846.
[28] LIN J, GAN C, HAN S. TSM: temporal shift module for efficient video understanding[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea: IEEE Press, 2019: 7082-7092.
[29] LIN Z Y, GENG S J, ZHANG R R, et al. Frozen CLIP models are efficient video learners[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2022: 388-404.
[30] LI X H, ZHU Y H, WANG L M. ZeroI2V: zero-cost adaptation of pre-trained transformers from image to video[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer. 2025: 425-443.
[31] XIE S N, SUN C, HUANG J, et al. Rethinking spatiotemporal feature learning: speed-accuracy trade-offs in video classification[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 318-335.
[32] ZHAO Z Y, HUANG B K, XING S, et al. Asymmetric masked distillation for pre-training small foundation models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, USA: IEEE Press, 2024: 18516-18526.

Please choose a citation manager

Content to export