Fatigue Driving Detection Based on Eye and Mouth State Recognition Network

doi:10.19678/j.issn.1000-3428.0064445

Abstract

Abstract: Driver fatigue is an important factor leading to traffic accidents，and it is particularly important to accurately detect the driving state of drivers. However，existing fatigue detection methods have the problems of high misjudgment rate and low robustness.To ensure high detection accuracy and improve the real-time performance，a fatigue driving detection method based on multi-index fusion and a state recognition network is proposed to deeply analyze the driver's fatigue state. An optimized single-stage face algorithm called RetinaFace is used to obtain the position of the face and locations of five signs. According to the coordinates of the key points of the eyes and corner of the mouth，the eyes and mouth area are rotated to the horizontal and intercepted，respectively. Existing datasets were reconstructed and classified to train the Eye and Mouth State Discrimination Network （EMSD-Net） based on a ghost module and then identify whether the eyes are open or closed and the mouth is yawning. Finally，according to the eye and mouth state，the Percentage of Eyelid Closure over the Pupil over Time （PERCLOS），Continuous Eye Closure Time （CCT），and Sustainable Yawn Time （SYT） are used to judge fatigue. The corresponding fatigue degree is obtained，which gives a more effective early warning effect. The experimental results on the new data set based on the NHTU-DDD，YawDD，and CEW datasets show that the proposed method has a fatigue feature recognition accuracy of 95.3%，an average single frame fatigue detection time of 32.6 ms，shows a low false positive rate，and a high real-time performance while ensuring detection accuracy.

Key words: fatigue driving detection, RetinaFace algorithm, state recognition, multi-feature fusion, Ghost module

摘要： 驾驶员疲劳驾驶是引发交通事故的重要因素，因此对驾驶员的驾驶状态进行精准检测尤为关键，然而现有的疲劳检测方法存在误判率高、鲁棒性低等问题。提出一种结合多特征融合与状态识别网络的疲劳驾驶检测方法，分析驾驶员的疲劳状态，利用优化后的单阶段人脸检测算法RetinaFace获取人脸位置及5个标志定位，根据双眼和嘴角关键点坐标将双眼及嘴部区域分别旋转至水平并截取。对现有数据集进行重新分类，用来训练以Ghost模块为基础的眼嘴状态识别网络（EMSD-Net），并对双眼开合状态及嘴部是否哈欠进行识别。最后，根据眼嘴状态，使用单位时间眼睛闭合的百分比、持续闭眼时间和持续哈欠时间为指标进行疲劳判断，并得出相应的疲劳程度，从而起到更有效的预警效果。在NHTU-DDD、YawDD和CEW数据集基础上构建的新数据集上的实验结果表明，所提方法的疲劳特征识别准确率为95.3%，单帧疲劳检测的平均时间为32.6 ms，具有较低的误判率，且在保证检测准确率基础上，有较高的实时性。

关键词: 疲劳驾驶检测, RetinaFace算法, 状态识别, 多特征融合, Ghost模块

CLC Number:

TP391.4

ZHANG Boyi, ZHE Tiantian, ZHAO Xinxu, LIU Qinghua, WANG Jiachen. Fatigue Driving Detection Based on Eye and Mouth State Recognition Network[J]. Computer Engineering, 2023, 49(5): 310-320.

张博熠, 者甜甜, 赵新旭, 刘庆华, 王家晨. 基于眼嘴状态识别网络的疲劳驾驶检测[J]. 计算机工程, 2023, 49(5): 310-320.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0064445

http://www.ecice06.com/EN/Y2023/V49/I5/310

Figures/Tables 20

References

[1] KOESDWIADY A,SOUA R,KARRAY F,et al.Recent trends in driver safety monitoring systems:state of the art and challenges[J].IEEE Transactions on Vehicular Technology,2017,66(6):4550-4563.
[2] LAWOYIN S.Novel technologies for the detection and mitigation of drowsy driving[EB/OL].[2022-03-10] https://www.zhangqiaokeyan.com/open-access_resources_thesis/0100049312789.html.
[3] CHAI M.Drowsiness monitoring based on steering wheel status[J].Transportation Research Part D:Transport and Environment,2019,66:95-103.
[4] TANGO F,BOTTA M.Real-time detection system of driver distraction using machine learning[J].IEEE Transactions on Intelligent Transportation Systems,2013,14(2):894-905.
[5] MIN J,XIONG C,ZHANG Y,et al.Driver fatigue detection based on prefrontal EEG using multi-entropy measures and hybrid model[J].Biomedical Signal Processing and Control,2021,69:57-65.
[6] ZHANG G Y,ETEMAD A.Capsule attention for multimodal EEG-EOG representation learning with application to driver vigilance estimation[J].IEEE Transactions on Neural Systems and Rehabilitation Engineering:a Publication of the IEEE Engineering in Medicine and Biology Society,2021,29:1138-1149.
[7] SATTI A T,KIM J,YI E,et al.Microneedle array electrode-based wearable EMG system for detection of driver drowsiness through steering wheel grip[J].Sensors (Basel,Switzerland),2021,21(15):91-102.
[8] VIOLA P,JONES M.Robust real-time face detection[C]//Proceedings of the 8th IEEE International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2004,57(2):137-154.
[9] ZHANG K P,ZHANG Z P,LI Z F,et al.Joint face detection and alignment using multitask cascaded convolutional networks[J].IEEE Signal Processing Letters,2016,23(10):1499-1503.
[10] DENG J K,GUO J,VERVERAS E,et al.RetinaFace:single-shot multi-level face localisation in the wild[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2020:5202-5211.
[11] 耿磊,袁菲,肖志涛,等.基于面部行为分析的驾驶员疲劳检测方法[J].计算机工程,2018,44(1):274-279. GENG L,YUAN F,XIAO Z T,et al.Driver fatigue detection method based on facial behavior analysisfull text replacement[J].Computer Engineering,2018,44(1):274-279.(in Chinese)
[12] ZHANG Z B,CHEN Y Z,YANG Y Z.Driver fatigue detection system based on machine vision[C]//Proceedings of the 7th World Congress on Intelligent Control and Automation.Washington D.C.,USA:IEEE Press,2008:3979-3984.
[13] JI Y Y,WANG S G,ZHAO Y,et al.Fatigue state detection based on multi-index fusion and state recognition network[J].IEEE Access,2019,7:64136-64147.
[14] 娄平,杨欣,胡辑伟,等.基于边缘计算的疲劳驾驶检测方法[J].计算机工程,2021,47(7):13-20,29. LOU P,YANG X,HU J W,et al.Fatigue driving detection method based on edge computing[J].Computer Engineering,2021,47(7):13-20,29.(in Chinese)
[15] HU J,SHEN L,SUN G.Squeeze-and-excitation networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:7132-7141.
[16] WANG Q L,WU B G,ZHU P F,et al.ECA-net:efficient channel attention for deep convolutional neural networks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2020:11531-11539.
[17] WANG Q,WU B,ZHU P,et al.ECA-Net:efficient channel attention for deep convolutional neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2020:11534-11542.
[18] HAN K,WANG Y H,TIAN Q,et al.GhostNet:more features from cheap operations[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition Seattle.Washington D.C.,USA:IEEE Press,2020:1577-1586.
[19] ACıOĞLU A,ERÇELEBI E.Real time eye detection algorithm for PERCLOS calculation[C]//Proceedings of the 24th Signal Processing and Communication Application Conference.Washington D.C.,USA:IEEE Press,2016:1641-1644.
[20] SONG F,TAN X,LIU X,et al.Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients[J].Pattern Recognition,2014,47(9):2825-2838.
[21] ABTAHI S,OMIDYEGANEH M,SHIRMOHAMMADI S,et al.YawDD:a yawning detection dataset[C]//Proceedings of the 5th ACM Multimedia Systems Conference.New York,USA:ACM,2014:24-28.
[22] WENG C H,LAI Y H,LAI S H.Driver drowsiness detection via a hierarchical temporal deep belief network[M]//CHEN C S,LU J W,MA K K.Computer Vision-ACCV 2016 Workshops.Berlin,Germany:Springer,2017:117-133.
[23] HE K M,ZHANG X Y,REN S Q,et al.Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2016:770-778.
[24] HE K M,ZHANG X Y,REN S Q,et al.Identity mappings in deep residual networks[M]//LEIBE B,MATAS J, SEBE N,et al.Computer Vision-ECCV 2016.Berlin,Germany:Springer,2016:630-645.
[25] HOWARD A G,ZHU M,CHEN B,et al.MobileNets:efficient convolutional neural networks for mobile vision applications[EB/OL].[2022-03-10].https://arxiv.org/abs/1704.04861.
[26] SANDLER M,HOWARD A,ZHU M L,et al.MobileNetV2:inverted residuals and linear bottlenecks[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:4510-4520.
[27] SZEGEDY C,VANHOUCKE V,IOFFE S,et al.Rethinking the inception architecture for computer vision[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2016:2818-2826.

Please choose a citation manager

Content to export