Research on Temporal-Spatial Semantic-Driven Progressive Multiview Action Debiasing

doi:10.19678/j.issn.1000-3428.0069307

Abstract

Abstract:

In practical applications, data collected from single-view cameras often lose visibility in certain areas due to object occlusion. Therefore, analyzing data from multiple views is crucial for maintaining social stability and public safety. To address the bias in multiview action recognition, arising from spatial semantic inconsistencies among different views and temporal semantic disparities during the execution of the same action, a multiview progressive debiasing method is proposed. First, guided by evidence theory within the context of multiple views for the same sample, the method leverages isomorphism across different views to mitigate inter-view bias. This involves optimizing the weights of features from different views to obtain a more comprehensive and unbiased representation. Second, employing a multi-granularity decoupling strategy, the method analyzes the impact of different granularities on debiased expression of features, thereby accurately separating relevant and irrelevant features while avoiding significant differences in representation caused by irrelevant information within a single view. Finally, the method constructs different feature weights along the temporal dimension, enhancing consistency in features within the same view and mitigating representation disparities for the same sample. The effectiveness of the proposed method is validated on multiple datasets, achieving cross-view accuracy rates of 97.4% and 96.4% on the N-UCLA and NTU-RGB+D datasets, respectively. This method not only meets the requirements for accurate recognition analysis under multiple views but also provides an effective solution to the bias problem in multiview recognition from a novel debiasing perspective.

Key words: multiview action recognition, progressive debiasing, evidence theory, decoupling, multi-granularity

摘要：

在实际应用中, 单视角摄像头采集数据由于物体存在遮挡而失去对某些区域的可见性, 因此结合多个视角下的数据进行行为分析对于维护社会稳定及民生安全至关重要。针对多视角行为识别中存在的偏置问题, 即不同视角下空间语义不一致导致的视角间行为表征差异以及同一行为执行过程中的时序语义不一致导致的行为表征差异, 提出一种渐进去偏置的多视角方法。首先, 在多视角下的同一行为样本中以证据理论为引导, 结合不同视角下的行为同构性进行视角间行为去偏置, 优化不同视角下关注的行为特征权重, 以获得更全面的无偏行为表示。其次, 结合多粒度解耦策略, 分析不同粒度对行为特征无偏表达的影响, 准确分离行为相关和行为无关特征, 以避免视角内行为无关信息扰乱行为表征导致的显著差异。最后, 在时序维度上构建不同行为特征权重, 增强同一视角内行为特征一致性, 减弱同一行为的行为表征差异。在多个数据集上的实验结果验证了所提方法的有效性, 在N-UCLA和NTU-RGB+D数据集上的跨视角准确率分别达到了97.4%和96.4%, 并且所提方法在满足多视角下对行为识别进行准确分析应用需求的同时通过一种新的去偏置思路为多视角行为识别问题提供了一种有效的解决方案。

关键词: 多视角行为识别, 渐进式去偏置, 证据理论, 解耦, 多粒度

ZHONG Xian, CHEN Liang, LIU Wenxuan, YE Shu, JIANG Kui, WANG Zheng, LIN Chia-Wen. Research on Temporal-Spatial Semantic-Driven Progressive Multiview Action Debiasing[J]. Computer Engineering, 2025, 51(1): 1-10.

钟忺, 陈亮, 刘文璇, 叶舒, 江奎, 王正, 林嘉文. 时空语义驱动的渐进多视角行为去偏置研究[J]. 计算机工程, 2025, 51(1): 1-10.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0069307

https://www.ecice06.com/EN/Y2025/V51/I1/1

Figures/Tables 8

Fig.1 Framework of progressive debiasing method

Fig.2 Examples of inter-view and intra-view biasing

Fig.3 Comparison of different head segmentation ratios

Fig.4 Comparison of different independence parameters

Fig.5 Confusion matrix visualization

References 49

1	刘思进, 朱小飞, 彭展望. 联合多任务学习的对话情感分类和行为识别. 计算机学报, 2023, 46 (9): 1947- 1960.
	LIU S J , ZHU X F , PENG Z W . Dialogue sentiment classification and act recognition based on multi-task learning. Chinese Journal of Computers, 2023, 46 (9): 1947- 1960.
2	蒲瞻星, 葛永新. 基于多特征融合的小样本视频行为识别算法. 计算机学报, 2023, 46 (3): 594- 608.
	PU Z X , GE Y X . Few-shot action recognition in video based on multi-feature fusion. Chinese Journal of Computers, 2023, 46 (3): 594- 608.
3	YOU H , ZHONG X , LIU W X , et al. Converting artificial neural networks to ultralow-latency spiking neural networks for action recognition. IEEE Transactions on Cognitive and Developmental Systems, 2024, 16 (4): 1533- 1545. doi: 10.1109/TCDS.2024.3375620
4	张洋, 姚登峰, 江铭虎, 等. 基于EfficientDet网络的细粒度吸烟行为识别. 计算机工程, 2022, 48 (3): 302-309, 314. URL
	ZHANG Y , YAO D F , JIANG M H , et al. Fine-grained smoking behavior recognition based on EfficientDet network. Computer Engineering, 2022, 48 (3): 302-309, 314. URL
5	闫兴亚, 匡娅茜, 白光睿, 等. 基于深度学习的学生课堂行为识别方法. 计算机工程, 2023, 49 (7): 251- 258. URL
	YAN X Y , KUANG Y Q , BAI G R , et al. Student classroom behavior recognition method based on deep learning. Computer Engineering, 2023, 49 (7): 251- 258. URL
6	ZHAO L, WANG Y X, ZHAO J P, et al. Learning view-disentangled human pose representation by contrastive cross-view mutual information maximization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2021: 12793-12802.
7	DHIMAN C , VISHWAKARMA D K . View-invariant deep architecture for human action recognition using two-stream motion and shape temporal dynamics. IEEE Transactions on Image Processing, 2020, 29, 3835- 3844. doi: 10.1109/TIP.2020.2965299
8	ZHONG X, ZHOU Z, LIU W X, et al. VCD: view-constraint disentanglement for action recognition[C]//Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Washington D.C., USA: IEEE Press, 2022: 2170-2174.
9	LIU W X , ZHONG X , ZHOU Z , et al. Dual-recommendation disentanglement network for view fuzz in action recognition. IEEE Transactions on Image Processing, 2023, 32, 2719- 2733. doi: 10.1109/TIP.2023.3273459
10	詹健浩, 甘利鹏, 毕永辉, 等. 基于知识蒸馏的多模态融合行为识别方法. 计算机工程, 2023, 49 (10): 280-288, 297. doi: 10.3778/j.issn.1002-8331.2201-0238
	ZHAN J H , GAN L P , BI Y H , et al. Action recognition method with multi-modality fusion based on knowledge distillation. Computer Engineering, 2023, 49 (10): 280-288, 297. doi: 10.3778/j.issn.1002-8331.2201-0238
11	施海勇, 侯振杰, 巢新, 等. 多模态时空特征表示及其在行为识别中的应用. 中国图象图形学报, 2023, 28 (4): 1041- 1055.
	SHI H Y , HOU Z J , CHAO X , et al. Multimodal spatial-temporal feature representation and its application in action recognition. Journal of Image and Graphics, 2023, 28 (4): 1041- 1055.
12	LI J N, WONG Y, ZHAO Q, et al. Unsupervised learning of view-invariant action representations[EB/OL]. [2024-01-02]. http://arxiv.org/abs/1809.01844.
13	SHAO Z P , LI Y F , ZHANG H . Learning representations from skeletal self-similarities for cross-view action recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31 (1): 160- 174. doi: 10.1109/TCSVT.2020.2965574
14	BAHRAMPOUR S , NASRABADI N M , RAY A , et al. Multimodal task-driven dictionary learning for image classification. IEEE Transactions on Image Processing, 2016, 25 (1): 24- 38. doi: 10.1109/TIP.2015.2496275
15	LIU Z G , WANG L , YIN Z Y , et al. Task-driven joint dictionary learning model for multi-view human action recognition. Digital Signal Processing, 2022, 126, 103487. doi: 10.1016/j.dsp.2022.103487
16	LI Y W, LI Y, VASCONCELOS N. RESOUND: towards action recognition without representation bias[C]//Proceedings of the European Conference on Computer Vision. Berlin, Germany: Springer, 2018: 520-535.
17	KIM B, KIM H, KIM K, et al. Learning not to learn: training deep neural networks with biased data[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2019: 9012-9020.
18	CHOI J, GAO C, MESSOU J C E, et al. Why can't I dance in the mall? Learning to mitigate scene bias in action recognition[EB/OL]. [2024-01-02]. http://arxiv.org/abs/1912.05534.
19	BAHNG H, CHUN S, YUN S, et al. Learning de-biased representations with biased representations[EB/OL]. [2024-01-02]. http://arxiv.org/abs/1910.02806.
20	BAO W T, YU Q, KONG Y. Evidential deep learning for open set action recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2021: 13349-13358.
21	DUAN H D, ZHAO Y, CHEN K, et al. Mitigating representation bias in action recognition: algorithms and benchmarks[C]//Proceedings of the European Conference on Computer Vision. Berlin, Germany: Springer, 2023: 557-575.
22	LI Q K, HUANG X L, LUO Y W, et al. Mitigating context bias in action recognition via skeleton-dominated two-stream network[C]//Proceedings of the 2023 Workshop on Advanced Multimedia Computing for Smart Manufacturing and Engineering. New York, USA: ACM Press, 2023: 65-70.
23	GRETTON A , BOUSQUET O , SMOLA A , et al. Measuring statistical dependence with Hilbert-Schmidt norms. Berlin, Germany: Springer, 2005.
24	SENSOY M, KAPLAN L, KANDEMIR M. Evidential deep learning to quantify classification uncertainty[EB/OL]. [2024-01-02]. http://arxiv.org/abs/1806.01768.
25	TIAN C W , ZHENG M H , ZUO W M , et al. A cross Transformer for image denoising. Information Fusion, 2024, 102, 102043. doi: 10.1016/j.inffus.2023.102043
26	TIAN C W , ZHENG M H , LI B , et al. Perceptive self-supervised learning network for noisy image watermark removal. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34 (8): 7069- 7079. doi: 10.1109/TCSVT.2024.3349678
27	WANG J, NIE X H, XIA Y, et al. Cross-view action modeling, learning, and recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2014: 2649-2656.
28	SHAHROUDY A, LIU J, NG T T, et al. NTU RGB+D: a large scale dataset for 3D human activity analysis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2016: 1010-1019.
29	CHEN Y X, ZHANG Z Q, YUAN C F, et al. Channel-wise topology refinement graph convolution for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2021: 13359-13368.
30	CHI H G, HA M H, CHI S, et al. InfoGCN: representation learning for human skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 20186-20196.
31	FEICHTENHOFER C, FAN H Q, MALIK J, et al. SlowFast networks for video recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2019: 6202-6211.
32	HARA K, KATAOKA H, SATOH Y. Can spatiotemporal 3D CNNs retrace the history of 2D CNNs and ImageNet? [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2018: 6546-6555.
33	KINGMA D P, BA J. Adam: a method for stochastic optimization[EB/OL]. [2024-01-02]. http://arxiv.org/abs/1412.6980.
34	NIE Q , WANG J L , WANG X , et al. View-invariant human action recognition based on a 3D bio-constrained skeleton model. IEEE Transactions on Image Processing, 2019, 28 (8): 3959- 3972. doi: 10.1109/TIP.2019.2907048
35	SI C Y, CHEN W T, WANG W, et al. An attention enhanced graph convolutional LSTM network for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2019: 1227-1236.
36	CHENG K, ZHANG Y F, HE X Y, et al. Skeleton-based action recognition with shift graph convolutional network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2020: 183-192.
37	LI L G, WANG M S, NI B B, et al. 3D human action representation learning via cross-view consistency pursuit[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2021: 4741-4750.
38	LIU X , LI Y , XIA R . Adaptive multi-view graph convolutional networks for skeleton-based action recognition. Neurocomputing, 2021, 444, 288- 300. doi: 10.1016/j.neucom.2020.03.126
39	GUO T Y, LIU H, CHEN Z, et al. Contrastive learning from extremely augmented skeleton sequences for self-supervised action recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2022: 762-770.
40	SONG Y F , ZHANG Z , SHAN C F , et al. Constructing stronger and faster baselines for skeleton-based action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45 (2): 1474- 1488. doi: 10.1109/TPAMI.2022.3157033
41	ZHANG J H, LIN L L, LIU J Y. Hierarchical consistent contrastive learning for skeleton-based action recognition with growing augmentations[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2023: 3427-3435.
42	VYAS S, RAWAT Y S, SHAH M. Multi-view action recognition using cross-view video prediction[C]//Proceedings of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 427-444.
43	XU C , WU X , LI Y C , et al. Cross-modality online distillation for multi-view action recognition. Neurocomputing, 2021, 456, 384- 393. doi: 10.1016/j.neucom.2021.05.077
44	PARK Y , WOO S , LEE S M , et al. Cross-modal alignment and translation for missing modality action recognition. Computer Vision and Image Understanding, 2023, 236, 103805. doi: 10.1016/j.cviu.2023.103805
45	ULLAH A , MUHAMMAD K , HUSSAIN T , et al. Conflux LSTMs network: a novel approach for multi-view action recognition. Neurocomputing, 2021, 435, 321- 329. doi: 10.1016/j.neucom.2019.12.151
46	PATRICK M , CAMPBELL D , ASANO Y , et al. Keeping your eye on the ball: trajectory attention in video Transformers. Advances in Neural Information Processing Systems, 2021, 34, 12493- 12506.
47	PIERGIOVANNI A J, RYOO M S. Recognizing actions in videos from unseen viewpoints[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2021: 4124-4132.
48	DAS S, RYOO M S. ViewCLR: learning self-supervised video representation for unseen viewpoints[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Washington D.C., USA: IEEE Press, 2023: 5573-5583.
49	CASELLA G , GEORGE E I . Explaining the Gibbs sampler. The American Statistician, 1992, 46 (3): 167- 174. doi: 10.1080/00031305.1992.10475878

[1]	ZHU Yanbin, WANG Runmin, CHEN Hua, CAO Xiaofei, ZHU Zhenlin, DING Yajun. Traffic Text Detection Method Based on Multi-Granularity Feature Enhancement Network [J]. Computer Engineering, 2024, 50(11): 80-88.
[2]	Wanhua LI, Yingjuan SUN, Yixuan LIU, Qian LIU. Session Recommendation Based on Global Graph and Multi-granularity Intention Unit [J]. Computer Engineering, 2023, 49(10): 136-144, 153.
[3]	WU Zhaoyuan, YU Zhengtao, HUANG Yuxin. Chinese-Vietnamese Cross-Lingual Word-Embedding Combined with Word Cluster Constraints [J]. Computer Engineering, 2023, 49(1): 82-91.
[4]	LIANG Dengyu, LIU Daming. Short Text Matching Model Combined with Multi-Granularity Information and External Knowledge [J]. Computer Engineering, 2022, 48(8): 129-135,143.
[5]	FU Jinwu, FAN Zizhu, SHI Linrui, GUO Xinyue, HUANG Yijing. Person Re-Identification Method Based on Multi-Scale and Multi-Granularity Fusion [J]. Computer Engineering, 2022, 48(3): 271-279.
[6]	LIU Xuan, CHI Mingmin. Scene Classification Research for Remote Sensing Images Based on Multi?Granularity Feature Distillation [J]. Computer Engineering, 2021, 47(1): 224-229,238.
[7]	SONG An, WANG Qin, GU Dawu, GUO Zheng, LIU Junrong, ZHANG Chi. FPGA-based Collection Method for Power Information of Clock Synchronization [J]. Computer Engineering, 2020, 46(6): 115-121.
[8]	FAN Qiaoling, JIA Xiangdong, JI Pengshan, LU Yi. Coverage Performance Comparison of DUDA and CUDA Schemes for Heterofeneous Network [J]. Computer Engineering, 2020, 46(3): 151-156.
[9]	WANG Chenxu, WANG Xiaochen, YU Dunhui, WU Shan. Software Crowdsourcing Task Decomposition Algorithm Based on Dynamic Decoupling [J]. Computer Engineering, 2019, 45(8): 120-124,134.
[10]	ZHI Hongxin,YU Hongtao,LI Shaomei,GAO Chao. Temporal pyramid construction method based on SRU [J]. Computer Engineering, 2019, 45(5): 243-248.
[11]	FAN Qiaoling, JIA Xiangdong, CHEN Yuwan, XU Wenjuan. Decoupling Cascaded Multilevel HetNet OMA and NOMA Schemes and Performance Comparison [J]. Computer Engineering, 2019, 45(11): 97-101,106.
[12]	WU Lian,MA Minyao,HUANG Yifeng,ZHAO Yong. Linux Virus Detection Study Based on AdaBoost Algorithm [J]. Computer Engineering, 2018, 44(8): 161-166,173.
[13]	WANG Ke,LI Peng,JIN Yu,LIU Yu. Dual-mode Map Matching Algorithm Based on Three Evidences DS Theory [J]. Computer Engineering, 2018, 44(5): 316-321.
[14]	GAO Chengzhen,MA Yongfei,SUN Zhanxian,KE Ximing. Suppress Strategy to On-chip Power Noise Based on Pre-placement Decoupling Capacitor [J]. Computer Engineering, 2018, 44(11): 56-61.
[15]	MENG Yuanyuan,XU Liancheng,REN Min,WANG Yanfei. D-S Evidence Fusion Method Based on High Conflict Correction [J]. Computer Engineering, 2018, 44(1): 79-83,90.

Please choose a citation manager

Content to export