Student Classroom Behavior Recognition Method Based on Deep Learning

doi:10.19678/j.issn.1000-3428.0065369

Abstract

Abstract:

Student classroom behaviors can directly reflect the quality of the class, whereby the analysis and evaluation of classroom behaviors through artificial intelligence and big data can help improve the quality of teaching. Traditional student classroom behavior recognition methods rely on teachers' direct observation of students or an analysis of student surveillance videos after class. This method is time-consuming, labor-intensive, and has low recognition rate, making it difficult to follow problems in the classroom and during exams in real time. This study proposes a posture recognition method based on deep learning BetaPose. The data enhancement technology is used to improve the robustness of the subsequent detection model. The improved YOLOv5 target detection algorithm is used to obtain the human detection frame. Based on the MobileNetV3 model, the lightweight posture recognition model is designed to improve the accuracy of posture recognition in crowded scenes. The keypoints of the human body thus obtained are input into the linear classifier with improved modeling and expression ability to determine the final behavior results. The experimental show that the proposed lightweight posture recognition model BetaPose had the highest average recognition accuracy of 82.6% for various parts of the human body, and the recognition rates for various behaviors in simple and crowded scenes are above 91% and 85%, respectively. Therefore, the proposed model can be effectively recognize multiple behaviors in the classroom.

Key words: computer vision, behavior recognition, object detection, posture recognition, deep learning

摘要：

学生课堂行为动作能够直接反映课堂质量，通过人工智能和大数据对课堂行为进行分析和评估，有助于提高教学质量。传统的学生课堂行为识别方法通过老师直接观察学生状态，或者是课后通过监控视频进行分析，该课堂行为识别方法耗时耗力且识别率较低，难以实时反映课堂以及考试中存在的问题。提出基于深度学习的姿态识别方法BetaPose。采用数据增强技术提高后续检测模型的鲁棒性，通过改进的YOLOv5目标检测算法得到人体检测框，并基于MobileNetV3模型设计轻量级姿态识别模型，提高在拥挤场景下的姿态识别准确率，将得到的人体关键点图输入到线性分类器中，获得最终行为结果，有效提高模型的建模和表达能力。实验结果表明，所提轻量级姿态识别方法BetaPose对人体各个部位的平均识别准确率最高可达82.6%，在简易和拥挤场景下对各种行为的识别率分别达到91%和85%以上，能够有效识别课堂的多种行为。

关键词: 计算机视觉, 行为识别, 目标检测, 姿态识别, 深度学习

Xingya YAN, Yaxi KUANG, Guangrui BAI, Yue LI. Student Classroom Behavior Recognition Method Based on Deep Learning[J]. Computer Engineering, 2023, 49(7): 251-258.

闫兴亚, 匡娅茜, 白光睿, 李月. 基于深度学习的学生课堂行为识别方法[J]. 计算机工程, 2023, 49(7): 251-258.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0065369

http://www.ecice06.com/EN/Y2023/V49/I7/251

Figures/Tables 12

References 25

1	ZHANG X L, YANG Z Q, CHEN T Y, et al. Cooperative sensing and wearable computing for sequential hand gesture recognition. IEEE Sensors Journal, 2019, 19(14): 5775- 5783. doi: 10.1109/JSEN.2019.2904595
2	XIE Y F, ZHANG S, LIU Y D. Abnormal behavior recognition in classroom pose estimation of college students based on spatiotemporal representation learning. Traitement Du Signal, 2021, 38(1): 89- 95. doi: 10.18280/ts.380109
3	WANG Z X, JIANG F, SHEN R M. An effective yawn behavior detection method in classroom[C]//Proceedings of Neural Information Processing Systems. Berlin, Germany: Springer, 2019: 430-441.
4	魏艳涛, 秦道影, 胡佳敏, 等. 基于深度学习的学生课堂行为识别. 现代教育技术, 2019, 29(7): 87- 91. URL
	WEI Y T, QIN D Y, HU J M, et al. The recognition of students' classroom behaviors based on deep learning. Modern Educational Technology, 2019, 29(7): 87- 91. URL
5	DANG Q, YIN J Q, WANG B, et al. Deep learning based 2D human pose estimation: a survey. Tsinghua Science and Technology, 2019, 24(6): 663- 676. doi: 10.26599/TST.2018.9010100
6	CAO Z, SIMON T, WEI S E, et al. Realtime multi-person 2D pose estimation using part affinity fields[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 7291-7299.
7	NOORI F M, WALLACE B, UDDIN M Z, et al. A robust human activity recognition approach using OpenPose, motion features, and deep recurrent neural network[C]//Proceedings of Scandinavian Conference on Image Analysis. Berlin, Germany: Springer, 2019: 299-310.
8	SUZUKI S, AMEMIYA Y, SATO M. Enhancement of gross-motor action recognition for children by CNN with OpenPose[C]//Proceedings of the 45th Annual Conference of the Industrial Electronics Society. Washington D. C., USA: IEEE Press, 2019: 5382-5387.
9	FANG H S, XIE S Q, TAI Y W, et al. RMPE: regional multi-person pose estimation[C]//Proceedings of International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 2334-2343.
10	李晨, 何明, 王勇, 等. 基于深度学习的视频行为识别技术综述. 计算机应用研究, 2022, 39(9): 2561- 2569. URL
	LI C, HE M, WANG Y, et al. Review of video action recognition technology based on deep learning. Application Research of Computers, 2022, 39(9): 2561- 2569. URL
11	ZHONG Z, ZHENG L A, KANG G L, et al. Random erasing data augmentation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S. l. ]: AAAI Press, 2020: 13001-13008.
12	DEVRIES T, TAYLOR G W. Improved regularization of convolutional neural networks with cutout[EB/OL]. [2022-06-20]. https://arxiv.org/pdf/1708.04552.pdf.
13	SINGH K K, LEE Y J. Hide-and-seek: forcing a network to be meticulous for weakly-supervised object and action localization[C]//Proceedings of International Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 3544-3553.
14	BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: optimal speed and accuracy of object detection[EB/OL]. [2022-06-20]. https://arxiv.org/abs/2004.10934.
15	JIAO S X, MIAO T, GUO H T. Image target detection method using the YOLOv5 algorithm[C]//Proceedings of 3D Imaging Technologies—Multidimensional Signal Processing and Deep Learning. Berlin, Germany: Springer, 2021: 323-329.
16	YU J H, JIANG Y N, WANG Z Y, et al. UnitBox: an advanced object detection network[C]//Proceedings of the 24th International Conference on Multimedia. New York, USA: ACM Press, 2016: 516-520.
17	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 740-755.
18	NEWELL A, YANG K, DENG J. Stacked hourglass networks for human pose estimation[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 483-499.
19	HOWARD A, SANDLER M, CHEN B, et al. Searching for MobileNetV3[C]//Proceedings of International Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 1314-1324.
20	INOCENCIO L A, PEREIRA A A, SUCUPIRA M C A, et al. Brazilian network for HIV drug resistance surveillance (HIV-BResNet): a new survey of HIV recent diagnosed individuals collected in 2007/8[C]//Proceedings of the 18th International HIV Drug Resistance Workshop. [S. l. ]: AAAI Press, 2009: 1-10.
21	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 6000-6010.
22	HUANG X, WANG X X, LV W Y, et al. PP-YOLOv2: a practical object detector[EB/OL]. [2022-06-20]. https://arxiv.org/abs/2104.10419v1.
23	CHEN D, SHEN H. MAOD: an efficient anchor-free object detector based on MobileDet. IEEE Access, 2020, 8, 86564- 86572. URL
24	CHENG B W, XIAO B, WANG J D, et al. HigherHRNet: scale-aware representation learning for bottom-up human pose estimation[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 5385-5394.
25	FEICHTENHOFER C, FAN H Q, MALIK J, et al. SlowFast networks for video recognition[C]//Proceedings of International Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 6201-6210.

输入	Operator	Exp size	# out	SE	NL	步长
N²×3	Conv2d, 3×3	—	16	—	HS	2
112²×16	bneck, 3×3	16	16	√	RE	2
56²×16	bneck, 3×3	72	24	—	RE	2
28²×24	bneck, 3×3	88	24	—	RE	1
28²×24	bneck, 5×5	96	40	√	HS	2
14²×40	bneck, 5×5	240	40	√	HS	1
14²×40	bneck, 5×5	240	40	√	HS	1
14²×40	bneck, 5×5	120	48	√	HS	1
14²×48	bneck, 5×5	144	48	√	HS	1
14²×48	bneck, 5×5	288	96	√	HS	2
7²×96	bneck, 5×5	576	96	√	HS	1
7²×96	bneck, 5×5	576	96	√	HS	1
1²×96	Conv2d, 1×1	—	1 024	—	HS	1
N²×3	Conv2d, 1×1	—	k	—	—	1

输入	Operator	Exp size	# out	SE	NL	步长
N²×3	Conv2d, 3×3	—	16	—	HS	2
112²×16	bneck, 3×3	16	16	√	RE	2
56²×16	bneck, 3×3	72	24	—	RE	2
28²×24	bneck, 3×3	88	24	—	RE	1
28²×24	bneck, 5×5	96	40	√	HS	2
14²×40	bneck, 5×5	240	40	√	HS	1
14²×40	bneck, 5×5	240	40	√	HS	1
14²×40	bneck, 5×5	120	48	√	HS	1
14²×48	bneck, 5×5	144	48	√	HS	1
14²×48	bneck, 5×5	288	96	√	HS	2
7²×96	bneck, 5×5	576	96	√	HS	1
7²×96	bneck, 5×5	576	96	√	HS	1
1²×96	Conv2d, 1×1	—	1 024	—	HS	1
N²×3	Conv2d, 1×1	—	k	—	—	1

方法	mAP @0.50:0.95	mAP @0.5	mAP @0.75
OpenPose	61.8	84.9	67.5
Detectron	67.0	88.0	73.1
AlphaPose	73.3	89.2	79.1
本文方法	74.1	92.9	82.1

方法	mAP @0.50:0.95	mAP @0.5	mAP @0.75
OpenPose	61.8	84.9	67.5
Detectron	67.0	88.0	73.1
AlphaPose	73.3	89.2	79.1
本文方法	74.1	92.9	82.1

方法	mAP @0.50:0.95
方法	Head	Shoulder	Elbow	Wrist	Hip	Knee	Ankle	平均值
OpenPose	91.2	87.6	77.7	66.8	75.4	68.9	61.7	75.6
Detectron	92.1	89.3	78.9	69.8	76.2	71.6	64.7	77.5
AlphaPose	91.3	90.5	84.0	76.4	80.3	79.9	72.4	82.1
本文方法	91.9	91.2	82.4	75.4	81.7	80.7	75.6	82.6

Please choose a citation manager

Content to export