基于深度学习的学生课堂行为识别方法

doi:10.19678/j.issn.1000-3428.0065369

摘要/Abstract

摘要：

学生课堂行为动作能够直接反映课堂质量，通过人工智能和大数据对课堂行为进行分析和评估，有助于提高教学质量。传统的学生课堂行为识别方法通过老师直接观察学生状态，或者是课后通过监控视频进行分析，该课堂行为识别方法耗时耗力且识别率较低，难以实时反映课堂以及考试中存在的问题。提出基于深度学习的姿态识别方法BetaPose。采用数据增强技术提高后续检测模型的鲁棒性，通过改进的YOLOv5目标检测算法得到人体检测框，并基于MobileNetV3模型设计轻量级姿态识别模型，提高在拥挤场景下的姿态识别准确率，将得到的人体关键点图输入到线性分类器中，获得最终行为结果，有效提高模型的建模和表达能力。实验结果表明，所提轻量级姿态识别方法BetaPose对人体各个部位的平均识别准确率最高可达82.6%，在简易和拥挤场景下对各种行为的识别率分别达到91%和85%以上，能够有效识别课堂的多种行为。

关键词: 计算机视觉, 行为识别, 目标检测, 姿态识别, 深度学习

Abstract:

Student classroom behaviors can directly reflect the quality of the class, whereby the analysis and evaluation of classroom behaviors through artificial intelligence and big data can help improve the quality of teaching. Traditional student classroom behavior recognition methods rely on teachers' direct observation of students or an analysis of student surveillance videos after class. This method is time-consuming, labor-intensive, and has low recognition rate, making it difficult to follow problems in the classroom and during exams in real time. This study proposes a posture recognition method based on deep learning BetaPose. The data enhancement technology is used to improve the robustness of the subsequent detection model. The improved YOLOv5 target detection algorithm is used to obtain the human detection frame. Based on the MobileNetV3 model, the lightweight posture recognition model is designed to improve the accuracy of posture recognition in crowded scenes. The keypoints of the human body thus obtained are input into the linear classifier with improved modeling and expression ability to determine the final behavior results. The experimental show that the proposed lightweight posture recognition model BetaPose had the highest average recognition accuracy of 82.6% for various parts of the human body, and the recognition rates for various behaviors in simple and crowded scenes are above 91% and 85%, respectively. Therefore, the proposed model can be effectively recognize multiple behaviors in the classroom.

Key words: computer vision, behavior recognition, object detection, posture recognition, deep learning

闫兴亚, 匡娅茜, 白光睿, 李月. 基于深度学习的学生课堂行为识别方法[J]. 计算机工程, 2023, 49(7): 251-258.

Xingya YAN, Yaxi KUANG, Guangrui BAI, Yue LI. Student Classroom Behavior Recognition Method Based on Deep Learning[J]. Computer Engineering, 2023, 49(7): 251-258.

https://www.ecice06.com/CN/Y2023/V49/I7/251

图/表 12

图1 基于深度学习的学生课堂行为识别方法

Fig.1 Student classroom behavior recognition method based on deep learning

图2 YOLOv5网络结构

Fig.2 Structure of YOLOv5 network

图3 在不同IoU、GIoU和DIoU损失函数条件下预测框与检测框的对比

Fig.3 Comparison of prediction frames and detection frames under different IoU, GIoU and DIoU loss function conditions

图4 人体关键点示意图

Fig.4 Schematic diagram of human keypoints

图5 Hourglass网络结构

Fig.5 Structure of Hourglass network

表1 MobileNetV3-SPPE网络结构

Table 1 Structure of MobileNetV3-SPPE network

输入	Operator	Exp size	# out	SE	NL	步长
N²×3	Conv2d, 3×3	—	16	—	HS	2
112²×16	bneck, 3×3	16	16	√	RE	2
56²×16	bneck, 3×3	72	24	—	RE	2
28²×24	bneck, 3×3	88	24	—	RE	1
28²×24	bneck, 5×5	96	40	√	HS	2
14²×40	bneck, 5×5	240	40	√	HS	1
14²×40	bneck, 5×5	240	40	√	HS	1
14²×40	bneck, 5×5	120	48	√	HS	1
14²×48	bneck, 5×5	144	48	√	HS	1
14²×48	bneck, 5×5	288	96	√	HS	2
7²×96	bneck, 5×5	576	96	√	HS	1
7²×96	bneck, 5×5	576	96	√	HS	1
1²×96	Conv2d, 1×1	—	1 024	—	HS	1
N²×3	Conv2d, 1×1	—	k	—	—	1

图6 本文方法流程

Fig.6 Procedure of the proposed method

表2 不同方法在MSCOCO数据集上的测试结果

Table 2 Test results among different methods on MSCOCO dataset %

方法	mAP @0.50:0.95	mAP @0.5	mAP @0.75
OpenPose	61.8	84.9	67.5
Detectron	67.0	88.0	73.1
AlphaPose	73.3	89.2	79.1
本文方法	74.1	92.9	82.1

表3 不同方法在MPII数据集上的测试结果

Table 3 Test results among different methods on MPII dataset %

方法	mAP @0.50:0.95
方法	Head	Shoulder	Elbow	Wrist	Hip	Knee	Ankle	平均值
OpenPose	91.2	87.6	77.7	66.8	75.4	68.9	61.7	75.6
Detectron	92.1	89.3	78.9	69.8	76.2	71.6	64.7	77.5
AlphaPose	91.3	90.5	84.0	76.4	80.3	79.9	72.4	82.1
本文方法	91.9	91.2	82.4	75.4	81.7	80.7	75.6	82.6

图7 不同方法的姿态识别结果对比

Fig.7 Posture recognition results comparison among different methods

图8 不同方法在简易场景下的识别准确率对比

Fig.8 Recognition accuracy comparison among different methods in simple scenes

图9 不同方法在拥挤场景下的识别准确率对比

Fig.9 Recognition accuracy comparison among different methods in crowded scenes

参考文献 25

1	ZHANG X L, YANG Z Q, CHEN T Y, et al. Cooperative sensing and wearable computing for sequential hand gesture recognition. IEEE Sensors Journal, 2019, 19(14): 5775- 5783. doi: 10.1109/JSEN.2019.2904595
2	XIE Y F, ZHANG S, LIU Y D. Abnormal behavior recognition in classroom pose estimation of college students based on spatiotemporal representation learning. Traitement Du Signal, 2021, 38(1): 89- 95. doi: 10.18280/ts.380109
3	WANG Z X, JIANG F, SHEN R M. An effective yawn behavior detection method in classroom[C]//Proceedings of Neural Information Processing Systems. Berlin, Germany: Springer, 2019: 430-441.
4	魏艳涛, 秦道影, 胡佳敏, 等. 基于深度学习的学生课堂行为识别. 现代教育技术, 2019, 29(7): 87- 91. URL
	WEI Y T, QIN D Y, HU J M, et al. The recognition of students' classroom behaviors based on deep learning. Modern Educational Technology, 2019, 29(7): 87- 91. URL
5	DANG Q, YIN J Q, WANG B, et al. Deep learning based 2D human pose estimation: a survey. Tsinghua Science and Technology, 2019, 24(6): 663- 676. doi: 10.26599/TST.2018.9010100
6	CAO Z, SIMON T, WEI S E, et al. Realtime multi-person 2D pose estimation using part affinity fields[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 7291-7299.
7	NOORI F M, WALLACE B, UDDIN M Z, et al. A robust human activity recognition approach using OpenPose, motion features, and deep recurrent neural network[C]//Proceedings of Scandinavian Conference on Image Analysis. Berlin, Germany: Springer, 2019: 299-310.
8	SUZUKI S, AMEMIYA Y, SATO M. Enhancement of gross-motor action recognition for children by CNN with OpenPose[C]//Proceedings of the 45th Annual Conference of the Industrial Electronics Society. Washington D. C., USA: IEEE Press, 2019: 5382-5387.
9	FANG H S, XIE S Q, TAI Y W, et al. RMPE: regional multi-person pose estimation[C]//Proceedings of International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 2334-2343.
10	李晨, 何明, 王勇, 等. 基于深度学习的视频行为识别技术综述. 计算机应用研究, 2022, 39(9): 2561- 2569. URL
	LI C, HE M, WANG Y, et al. Review of video action recognition technology based on deep learning. Application Research of Computers, 2022, 39(9): 2561- 2569. URL
11	ZHONG Z, ZHENG L A, KANG G L, et al. Random erasing data augmentation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S. l. ]: AAAI Press, 2020: 13001-13008.
12	DEVRIES T, TAYLOR G W. Improved regularization of convolutional neural networks with cutout[EB/OL]. [2022-06-20]. https://arxiv.org/pdf/1708.04552.pdf.
13	SINGH K K, LEE Y J. Hide-and-seek: forcing a network to be meticulous for weakly-supervised object and action localization[C]//Proceedings of International Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 3544-3553.
14	BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: optimal speed and accuracy of object detection[EB/OL]. [2022-06-20]. https://arxiv.org/abs/2004.10934.
15	JIAO S X, MIAO T, GUO H T. Image target detection method using the YOLOv5 algorithm[C]//Proceedings of 3D Imaging Technologies—Multidimensional Signal Processing and Deep Learning. Berlin, Germany: Springer, 2021: 323-329.
16	YU J H, JIANG Y N, WANG Z Y, et al. UnitBox: an advanced object detection network[C]//Proceedings of the 24th International Conference on Multimedia. New York, USA: ACM Press, 2016: 516-520.
17	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2014: 740-755.
18	NEWELL A, YANG K, DENG J. Stacked hourglass networks for human pose estimation[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 483-499.
19	HOWARD A, SANDLER M, CHEN B, et al. Searching for MobileNetV3[C]//Proceedings of International Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 1314-1324.
20	INOCENCIO L A, PEREIRA A A, SUCUPIRA M C A, et al. Brazilian network for HIV drug resistance surveillance (HIV-BResNet): a new survey of HIV recent diagnosed individuals collected in 2007/8[C]//Proceedings of the 18th International HIV Drug Resistance Workshop. [S. l. ]: AAAI Press, 2009: 1-10.
21	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 6000-6010.
22	HUANG X, WANG X X, LV W Y, et al. PP-YOLOv2: a practical object detector[EB/OL]. [2022-06-20]. https://arxiv.org/abs/2104.10419v1.
23	CHEN D, SHEN H. MAOD: an efficient anchor-free object detector based on MobileDet. IEEE Access, 2020, 8, 86564- 86572. URL
24	CHENG B W, XIAO B, WANG J D, et al. HigherHRNet: scale-aware representation learning for bottom-up human pose estimation[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 5385-5394.
25	FEICHTENHOFER C, FAN H Q, MALIK J, et al. SlowFast networks for video recognition[C]//Proceedings of International Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 6201-6210.

[1]	牛瑞婷, 严天峰, 高锐, 王映植. 低信噪比下基于深度学习TCNN-MobileNet的调制识别[J]. 计算机工程, 2024, 50(7): 204-215.
[2]	肖慈, 徐杨, 张永丹, 冯明文, 黄易仟. 结合注意力和低光增强的夜间语义分割[J]. 计算机工程, 2024, 50(7): 271-281.
[3]	张诗婧, 莫绪涛, 赵行, 董杨林. 基于球面折反射成像和YOLOv7的内螺纹缺陷检测[J]. 计算机工程, 2024, 50(7): 282-292.
[4]	贵向泉, 刘世清, 李立, 秦庆松, 李唐艳. 基于改进YOLOv8的景区行人检测算法[J]. 计算机工程, 2024, 50(7): 342-351.
[5]	徐明亮, 李芳媛, 马浩然, 何飞. 大规模神经记录的峰电位聚类算法(特邀)[J]. 计算机工程, 2024, 50(6): 1-34.
[6]	魏琢艺, 罗迈, 李文兵, 曾远松, 余伟江, 杨跃东. 基于多源域适应的单细胞智能分类[J]. 计算机工程, 2024, 50(6): 48-55.
[7]	李子杰, 周菊香, 韩晓瑜, 甘健侯, 鹿泽光, 王俊. 序列特征与学习过程融合的知识追踪模型[J]. 计算机工程, 2024, 50(6): 77-85.
[8]	李永飞, 李铭洋, 常鑫, 曹可欣. 基于可解释性深度学习的物联网水质监测数据异常检测[J]. 计算机工程, 2024, 50(6): 179-187.
[9]	李致金, 汤佳辉, 闫金凤. 基于边缘计算的轻量化识别方法[J]. 计算机工程, 2024, 50(6): 287-295.
[10]	李雪, 王雅文, 张前进. 基于信息检索的源代码自动命名[J]. 计算机工程, 2024, 50(6): 304-310.
[11]	刘仕兵, 周诗涵. 高铁接触网绝缘子检测算法研究[J]. 计算机工程, 2024, 50(5): 200-208.
[12]	孙帮勇, 马铭, 于涛. 基于区域特征强化的多尺度伪装目标检测方法[J]. 计算机工程, 2024, 50(5): 209-219.
[13]	郑娟毅, 张庆珏, 董嘉豪, 郭梦月, 杨溥江. 一种深度学习的波束空间信道估计算法[J]. 计算机工程, 2024, 50(5): 298-305.
[14]	赵逸飞, 张俊华. 虚拟脊柱侧凸病例腰椎自动生成方法[J]. 计算机工程, 2024, 50(5): 323-329.
[15]	刘仲民, 严梁. 融合动态特征与注意力的敦煌壁画修复模型[J]. 计算机工程, 2024, 50(5): 342-353.

选择文件类型/文献管理软件名称

选择包含的内容