Integrate memory mechanism in multi-granularity deep framework for driver drowsiness detection

2.1 Traditional driver drowsiness detection methods

Driver drowsiness detection is becoming a hot topic in Advanced Driver Assistant Systems. Many traditional methods are applied to deal with this problem. The change of pupil diameter was utilized by Shirakata et al.to detect imperceptible drowsiness, which is effective but not convenient for a driver to take the equipment^[14]. Nakamura et al.utilized face alignment to estimate the degree of drowsiness via K-Nearest Neighbors (k-NN), which cannot achieve online performance^[8]. Spatial-temporal features for driver drowsiness detection were proposed by Akrout et al.^[10]. However, these features, based on Hough transformation, cannot work well in practical driving environments^[15] proposed a method for detecting driver drowsiness based on time-series analysis of the steering wheel angular velocity. Their approach involves using a temporal detection window to determine the steering wheel angular velocity over a time series, during which specified indicators of driver drowsiness become evident. Besides, the representations used in those methods are hand-crafted, which may not be flexible enough to adapt to complex situations faced in driving. In contrast, our method automatically learns facial representations, which is more effective for practical tasks.

In earlier research, to estimate the level of drowsiness, the measures to be focused on are single measures, such as vehicle-based, physiological, behavioral, or subjective measures. Researchers have reported good results using numerous less intrusive techniques to detect the drowsiness of drivers, including eyelid movement and gaze or head movement monitoring. Rumagit et al.investigated the relationship between the drowsiness and physiological conditions by utilizing an eye gaze tracker and the Japanese version of the Karolinska sleepiness scale within the driving simulator environment^[16]. Amirudin et al.analyzed two single measures that include physiological and behavioral measures, such as EEG signals and video sequences^[17]. Chmielińska et al.introduced an approach using CNNs and transferring learning techniques^[18]. The paper presents the results of scientific investigations aimed at developing detectors of the selected driver fatigue symptoms based on face images.

2.2 Driver drowsiness detection methods: CNN and RNN-based approaches

Deep learning approaches, such as CNN, have achieved success in representing information on images ^[19–21] and are widely used in the field of machine learning ^[22]. The use of CNN models for image classification can avoid the problem of high complexity and difficulty in feature extraction found in traditional classification methods and is, therefore, increasingly applied in facial recognition^[23]. Compared to traditional image classification methods, deep learning methods can use a large number of datasets for training and learn the best features to represent these data, making them more responsive to changes in the real world ^[24].

Recently, many researchers have also applied CNN to driver drowsiness detection. Park et al.combined the results of three existing networks by Support Vector Machines (SVM) to present the categories of videos^[25]. Later^[26–29], have also introduced various improved CNNs to capture facial regions under complex driving conditions to classify videos. However, those models can only classify videos into different categories; they cannot detect driver drowsiness online. In contrast, 3D-CNN is applied to extract spatial and temporal information by Yu et al.^[11]. However, the method can only capture features with a fixed temporal window. The above two methods utilize global face images, which cannot flexibly configure those patches containing the majority of drowsy information. Moreover, they struggle to capture dependencies with variable temporal lengths.

Due to the strong performance of LSTM Networks on sequential data ^[30–32], an increasing number of researchers propose combinations of CNNs and LSTMs to learn spatial and temporal representations of sequential frames. It is interesting that Liang et al.came up with convolutional layers with intra-layer recurrent connections to integrate the context information for object recognition^[33]. Donahue et al.provided a method that extracts visual features from images by CNN and learns the long-term dependencies from sequential data by LSTMs^[34]. In particular, the approach of Wang et al.and Jeong et al.processes images with CNN and models sequential labels by LSTMs concurrently and then combines the two representations through projection layers^{[35, 36]}. However, none of the above methods apply a multi-granularity approach to concentrate representations on important parts and flexibly fuse configurations of different regions.

2.3 Multi-granularity methods

Fine-grained methods mostly rely on object detection^[37–39], classifying all regions after identifying areas that may contain objects. Coarse-grained methods extract and encode overall image features through convolutional networks^[40] or vision transformers^[41], which can eliminate the interference of fine details, but their performance is often inferior to fine-grained methods. The multi-granularity methods combine coarse and fine granularity to capture discriminative spatial and temporal information at different semantic levels^[42].

Recently, multi-granularity methods have achieved several excellent results in some applications of computer vision. Li et al.proposed a temporal multi-granularity approach on action recognition^[43]. Their method achieved the state-of-the-art performance on action benchmarks but could not capture detailed appearance clues and local-to-global spatial information. Chen et al.applied multi-scale patches based on face alignment on face recognition^[44]. Wang et al.utilized multi-granularity regions detected by three granularities of CNN to generate a multi-granularity descriptor for fine-grained categorization, but this method cannot process sequential frames^[45]. Huang et al.proposed a multi-granularity extraction sub-network that extracts more efficient multi-granularity features while compressing the network parameters^[46]. They also included a feature rectification sub-network and a feature fusion sub-network to adaptively recalibrate and fuse the multi-granularity features. Finally, an LSTM network is applied to distinguish actions with similar appearances. However, this method cannot prioritize the most significant regions to get the most precise result and speed up the inference. Different from the approaches mentioned above, our method can capture both spatial multi-granularity information and long-term temporal dependencies. Particularly, our MCNN can learn representations on the most significant regions from well-aligned multi-granularity patches, and the proposed method has achieved the state-of-the-art accuracy on the National Tsing Hua University Driver Drowsiness Detection (NTHU-DDD) ^[47] dataset for driver drowsiness detection.

3.1 Well-aligned multi-granularity patches

It is well known that drowsy information is focused on several main facial parts, such as eyes, nose, and mouth. Alignment provides an excellent way to extract well-aligned features over frames, which effectively represent facial drowsy states. Besides, a global patch provides rough information for estimating a driver’s head and full face states, which assists in the decision of the driver’s drowsiness when the locations of parts are imprecise. Our method takes advantage of local regions and the global face at the same time.

We utilize face alignment technology to locate facial shape points. Given an image $$ I^t $$ with a face in the $$ t $$-th frame, we detect landmark points of facial shape $$ S^t $$ via regressing local binary features proposed by Ren et al.^[48]. From those points, it is convenient to get the locations of the main parts and important local regions. According to center points and specific sizes of all regions, we crop those patches from the original image and resize them into the same size, which are the well-aligned multi-granularity patches as the input of the CNN.

Those patches, including local regions, main parts, and the global face, are produced by three different mappings. As shown in Figure 3, a mapping $$ {\bf{\Phi}}^M_p $$ can select center points of facial features, such as eyes, nose, and mouth, from the facial shape $$ S^t $$ and crop patches of those parts from the input image $$ I^t $$ with given sizes $$ s_p $$. The mapping still needs to convert these patches into a unified size $$ s_u $$. Thus, the single-granularity patches of those main parts $$ {\bf{I}}_p^t $$ are generated. The operations of mappings $$ {\bf{\Phi}}^M_l $$ and $$ {\bf{\Phi}}^M_g $$ are similar to $$ {\bf{\Phi}}^M_p $$, while the differences lie in the locations and sizes of regions. The mapping $$ {\bf{\Phi}}^M_l $$ selects the corners of the eyes, mouth, and the sides of the nose as the interest of regions with size $$ s_l $$ and output local patches $$ {\bf{I}}_l^t $$. A global facial region with size $$ s_g $$ is chosen by the mapping $$ {\bf{\Phi}}^M_g $$, which finally produces a global facial patch $$ {\bf{I}}_g^t $$. Formally, the mappings are represented as

Figure 3. The procedure of extracting multi-granularity facial patches, which include three granularities: the main parts, local regions, and the global face.

By processing the input image $$ I^t $$ through the three mappings, we can obtain a set of well-aligned patches $$ {\bf{I}}_c^t $$ consisting of the main parts, local regions, and global face, which is represented as

where $$ {\bf{I}}^t_{i, :}, i\in\{l, p, g\} $$ represents all elements of a patch set $$ {\bf{I}}^t_i $$.

Compared to the original image, the patch set $$ {\bf{I}}_c^t $$, including both detailed appearance clues of parts and rough information of the full face, has more advantages in describing the facial states. Meanwhile, the relations between local and global regions are implied, which are the basis of mining useful features. Therefore, we take the set of patches $$ {\bf{I}}_c^t $$ as the input of CNN to learn effective representations.

3.2 Learning facial representations

Our approach learns representations by CNN but is not hand-crafted for its good performance in learning spatial features. We apply several convolutional layers to process each one in the set of patches $$ {\bf{I}}_c^t $$ independently. To fuse the information of all patches, a fully connected layer is arranged after all convolutional operations, which generates $$ N $$-dimensional descriptors combining local and global clues.

Every patch needs to be processed by convolutional operations at first. For a patch $$ I_{c, k}^t $$, the $$ k $$-th one of patch set $$ {\bf{I}}_c^t $$ with length $$ L $$, three convolutional layers are utilized to capture the spatial features, as shown in Figure 4. The first one is made with convolution and Rectified Linear Unit (ReLU) activation followed by a max-pooling operation, which projects a normalized 3-channel image to a higher dimensional representation. Only convolution and ReLU activation are selected in the second layer to enlarge the dimension of representation sequentially. The structure of the third convolutional layer is similar to the first layer but with different parameters to decrease the dimension. A representation $$ {\bf{x}}_k^t $$ of the patch $$ {\bf{I}}_{c, k}^t $$ can be generated by a mapping $$ {\bf{\Phi}}^C $$ consisting of those convolutional layers with parameters $$ {\mathit{\boldsymbol{\theta}}}^C_k $$, which is represented as

Figure 4. The three layers to capture the spatial features. The first layer is to project a normalized three-channel image onto a higher dimensional representation; the second layer is to enlarge the dimension of the representation; the third layer is to decrease the dimension with different parameters to the first layer.

where $$ {\mathit{\boldsymbol{\theta}}}^C_k $$ is the $$ k $$-th element of the convolutional parameter set $$ {\mathit{\boldsymbol{\theta}}}^C $$.

A fully connected layer is utilized to combine those representations extracted by the mapping $$ {\bf{\Phi}}^C $$ from the set of patches. But before combining operation, we concatenate those representations into a long vector $$ {\bf{x}}_c^t $$, formed as

With a specific weighted matrix $$ {\bf{W}}^C_f $$ and bias vector $$ {\bf{b}}^C_f $$, the combining $$ N $$-dimensional representation $$ {\bf{x}}^t $$ can be represented by the fully connected layer as

in which $$ {\bf{0}} $$ is a zero vector.

The descriptor $$ {\bf{x}}^t $$ contains not only detailed appearance information implied in every part but also the constrained relations between local regions and the global face. The effectiveness of the descriptor can be improved by appropriate objective functions and proper training methods.

Driver drowsiness detection is a binary classification problem; thus, the state of an input frame is just drowsy or not. We label drowsiness with 1 as the positive sample and normal state with 0 as the negative sample. A label $$ c $$ is expressed with a one-hot vector $$ {\bf{y}}_c $$, where, for example, a vector [0, 1] means the positive label.

To train the parameters of the CNN, we project the representation $$ {\bf{x}}^t $$ into the probabilities of each category $$ c\in\{0, 1\}, $$ by another fully connected layer with weights $$ {\bf{W}}^C_p $$, a bias vector $$ {\bf{b}}^C_p $$, and the probability vector $$ {\bf{p}}(c\|{\bf{x}}^t, {\bf{W}}^C_p, {\bf{b}}^C_p) $$ is normalized via a softmax layer. The cross-entropy, which can indicate the correct rate of classification, is selected as the objective function, and we utilize the Adam optimizer to train the whole CNN. The visual representations can also be generated by the convolutional layers and the first fully connected layer.

3.3 Exploring dynamical characteristics

The representation $$ {\bf{x}}^t $$ is extracted in a frame, while whether a driver is drowsy is judged by a certain period. We apply LSTMs to model the temporal dynamical characteristics of spatial representations on driver drowsiness detection.

An LSTM block consists of an input gate, a forget gate, an output gate, and a memory cell. Because of the three gates, the LSTM block can learn long-term dependencies in sequential data, and its parameters are easier to train. The memory cell can store long-term information in its vector, which can be rewritten or operated on in the next time steps. Besides, the number of hidden units should be chosen according to the dimension of the input representation $$ {\bf{x}}^t $$.

We employ multiple-layer LSTMs to mine the temporal features for driver drowsiness. A mapping $$ {\bf{\Phi}}^R $$ containing three-layer LSTMs with parameters $$ {\mathit{\boldsymbol{\theta}}}^R $$ is utilized to explore temporal clues of the representation $$ {\bf{x}}^t $$ generated by MCNN extractors and presents the hidden states $$ {\bf{h}}^t_3 $$ of the third layer as a representation containing temporal dependencies, which is represented as:

where $$ {\bf{h}}^{t-1}=\{{\bf{h}}^{t-1}_1, {\bf{h}}^{t-1}_2, {\bf{h}}^{t-1}_3\} $$ is the parameter set of these LSTM blocks in the last step.

A fully connected layer with weight $$ {\bf{W}}^R $$ and a bias vector $$ {\bf{b}}^R $$ is used to project the output of the mapping $$ {\bf{\Phi}}^R $$ into a two-dimensional vector that is then decoded by softmax operation to the probabilities $$ {\bf{p}}(c\|{\bf{h}}^t_3, {\bf{W}}^R, {\bf{b}}^R) $$ of the two categories. To solve the parameters, we take advantage of the Adam optimizer to train the LSTMs with a cross-entropy objective function.

The label $$ y^t $$ of the current frame can be predicted as the class with the maximum probability.

Similarly, the labels $$ {\bf{y}} $$ of the sequential data can be obtained.

4.1 Dataset

NTHU-DDD Dataset: The NTHU-DDD dataset includes five scenarios listed as glasses, no glasses, glasses at night, no glasses at night, and sunglasses. The training set involves 18 volunteers consisting of ten men and eight women who act as drivers with four different states in every scenario, while the evaluation set has four volunteers, including two men and two women. Non-sleepy videos contain only normal states, while sleepy videos combine normal and drowsy states together. Besides, blinking with nodding and yawning videos only record drowsy eyes and mouth, respectively. The NTHU-DDD dataset offers four annotation files recording the states of drowsiness, eyes, head, and mouth for every video. Table 1 gives the labels of drowsiness and three main parts.

Table 1

The labeled states of each part on the NTHU-DDD dataset

	0	1	2
Drowsiness	Normal	Drowsy	-
Eyes	Normal	Sleepy	-
Head	Normal	Nodding	Looking aside
Mouth	Normal	Yawning	Talking and laughing

It is worth emphasizing that the labels on the NTHU-DDD dataset are long-term memory, which means that the states of a frame may depend on the frames in the previous several seconds.

FI-DDD Dataset: A problem comes due to the long-term memory in NTHU-DDD, which is that a driver would still receive the warning prompts even if he had revised his drowsy state to normal for a few seconds. At the same time, those labels are unable to locate the drowsy states with high precision in the temporal dimension. To solve these problems, we relabel those videos with an instant principle, which means the latency is limited to 0.5 s, namely 15 frames for 30 FPS videos. Those typical states, such as closing eyes, yawning, and lowering head, are still considered as one of the pieces of evidence to judge whether a frame is drowsy. Those videos are cut into several clips that contain only the drowsy or normal states, alternatively according to our labels. To describe the transitional states between the normal and the drowsy, we reserve ten normal frames at the head and the tail of every clip with drowsiness. We name the relabeled dataset FI-DDD, which includes 14 drivers on the train set and four ones on the test set, as shown in Figure 5. The train set of FI-DDD in the daytime has 157 clips, and the test set has 92 clips, while in night scenarios, the train set has 126 clips, and the test set has 75 clips with about 530 frames on average.

Figure 5. FI-DDD Dataset.

Static image set: To train the parameters of CNN and analyze the effects of several factors, we build a static image set by sampling lots of frames from the FI-DDD dataset. The samples on the image set are labeled with drowsiness or normality, and the labels can almost indicate the true states of the corresponding images, even if a small number of images are matched with wrong labels due to a lack of temporal dependence. The static image set has 7, 498 images in the daytime; the train set includes 5, 239 images, and the test set has 2, 259 images. It has 2, 653 images in night scenarios; the train set includes 1, 750 images, and the test set has 903 images.

4.2 Implementation details

Face Alignment: We apply face alignment technology to locate those facial shape points for all videos. Face detection and tracking are combined to increase detecting rates and provide more accurate positions for faces on videos. Face alignment algorithms are based on those face positions. The face detector is from OpenCV, and the approach of face tracking is proposed by Danelljan et al.^[49]. We implement the method of Ren et al., retrain the model, and preprocess all videos to obtain the 51 landmark points for every frame^[48]. Those frames with no face will be recognized as empty and filled with zero coordinates for landmark points.

Multi-granularity: We obtain Multi-granularity patches considering two factors: different positions and sizes. We design to choose 15 positions from facial shape points, which are divided into three granularities: one global face with size $$ s_g=(160 \times 160) $$, four main parts with size $$ s_p=(64 \times 64) $$, and ten local regions with size $$ s_l=(32 \times 32) $$. The specific locations of all patches are shown in Figure 3. Before being sent to CNN, those patches are resized to size $$ s_u=(64 \times 64) $$, normalized to [-0.5, 0.5], and converted to 3 channels to ensure that our framework can process RGB data.

Dataset Usage: A static image set required for training the CNN parameters is sampled from the videos of FI-DDD with a specific frame interval. The result of CNN is directly related to multi-granularity patches and CNN parameters; we, thus, analyze the effects of those factors on the static image set, while all experiments for analyzing the effects of LSTM parameters are carried out on the FI-DDD dataset. To compare with the previous methods, we evaluate the proposed method on the evaluation set of the NTHU-DDD dataset.

4.3 Experimental analysis

As shown in Figure 6, the spatial and temporal features are extracted to detect driver drowsiness. The detection results of a driver under normal conditions are shown in Figure 6A, and the detection result of driver drowsiness is shown in Figure 6B, with the detected features marked with red color. In the experiments, there are two kinds of videos with different camera positions, and the proposed methods can work effectively for both.

Figure 6. The examples of driver drowsiness detection. (A) The normal status of a driver; (B) The detection result of driver drowsiness, while the detected features are marked with red colors.

To further explain the effects of alignment, multi-granularity, and CNN extractors, several groups of experiments are conducted on the static image set. We also provide experiments on the FI-DDD dataset to verify the effectiveness of LSTMs for detecting drowsiness on videos.

4.3.1 The importance of alignment

It is essential to carry out experiments to explain the significance of alignment and the effects of locating precision.

None-alignment vs. With Alignment: We provide another two none-alignment methods to sample those multi-granularity patches in facial bounding boxes: Uniform Sampling (US) and Specific Sampling (SS). The corresponding sizes of our Aligned sampling (AS) method and the two none-alignment ones are the same. Figure 7 (Left) shows the comparison of AS, US, and SS. AS considering alignment achieves the best accuracy at 87.4% on the test set of the static image set, which is 4.9% higher than the SS method and 6.2% higher than US. In conclusion, the alignment of facial patches, providing aligned representations, is an effective way to improve the accuracy of driver drowsiness detection.

Figure 7. Left: The comparison of different sampling methods, sampling over US, sampling SS, and our proposed sampling with AS; Right: The effect of alignment precision, $$ \sigma $$ is the standard deviation of normal distribution. The results (Acc) are achieved by CNN on the test set of the static image set.

Effects of alignment precision: We evaluate the effects of the alignment precision and investigate the influence quantitatively by adding random noise with a Gaussian distribution $$ N(0, \sigma) $$ over the well-aligned facial points. Figure 7 (Right) shows the results of a test set of the static image set, from which we discover that the accuracy decreases with the increasing standard deviation of noise and even less than 80% if $$ \sigma\geq10 $$ px. While the accuracy is more than 83% with $$ \sigma $$ less than 5 px, we make a conclusion that the proposed MCNN is robust to the corrupted locations if $$ \sigma\leq5 $$ px.

4.3.2 The effects of multi-granularity patches

Multi-granularity patches consist of local regions, main parts, and the global face. It is significant to conduct experiments and explain the importance of those granularities on driver drowsiness detection. We apply a fully connected layer and softmax operation to classify representations presented by MCNN extractors and analyze the effects of multi-granularity patches by the results of the classification.

Learning curve on different granularities: We take four different granularities, listed as local regions, main parts, the global face, and their combination, into account to analyze the effects of multi-granularity facial patches. Figure 8 illustrates the comparisons of those granularities, from which we know that the convergent speed of the method with global face granularity is the slowest compared with the others and that of local regions is the fastest, while the multi-granularity method achieves good performance on both convergent speed and accuracy. Aligned points can achieve higher precision in those local regions with abundant boundary texture, which results in more aligned representations and easier classification. Nevertheless, multi-granularity patches containing more aligned information are more effective in driver drowsiness detection.

Figure 8. The comparison of different granularities, the global face, main parts, local regions, and multi-granularity patches. The Curve of Acc over training times is achieved by CNN with different granularities on the test set of the static image set.

Effects of positions and sizes: We change the positions and sizes of facial patches, respectively. As shown in Figure 9 (Left), the main facial parts, including eyes, nose, and mouth, obtain the best accuracy 83.6% compared with the other single-granularity method. Obviously, the combination of those three granularities achieves the best accuracy at 87.4%. A conclusion comes that the most effective representation is extracted from the three main facial parts, while the fusion of local and global clues is an excellent way to obtain better facial representations.

Figure 9. Left: The comparison of patches with different positions, GF-the global face, MP-main parts (eyes, nose, and mouth), and LR-local regions(the corner of eyes, the sides of the nose, and the boundary of the mouth); Right: The comparison of patches with different sizes at all locations. Mg represents multi-granularity patches.

We set their sizes as the same and change the sizes to examine the difference between single-size and multi-granularity methods while keeping the patch locations constant. Figure 9 (Right) shows different regions with different sizes achieve 2.3% accuracy more than that of those single-size patches. The phenomenon is a result of the variation in sizes among different physiological parts; for example, the size of the global face is larger than that of a single eye. The above analysis presents that the multi-granularity method is an effective way to represent facial features.

4.3.3 The parameters selection of MCNN extractor

The structure parameters of the convolutional layers are listed in Table 2, which are the same in all parallel convolutional paths. A patch with size $$ 64 \times64 $$ processed by those convolutional layers is projected to a tensor with size $$ 16 \times 16 \times 4 $$. A representation of the patch is generated by reshaping the tensor to a 1024-dimensional vector, which is the input of a fully connected layer.

Table 2

The parameters of the three convolutional layers

Layers	Operations	Attributions
1st	Convolution	Size: $$ [5\times5\times3\times32] $$
	Activation	ReLU
	Max pooling	Strides: $$ [2\times2] $$
2nd	Convolution	Size: $$ [5\times5\times32\times64] $$
	Activation	ReLU
	Max pooling	Not Used
3rd	Convolution	Size: $$ [5\times5\times64\times4] $$
	Activation	ReLU
	Max pooling	Strides: $$ [2\times2] $$

A fully connected layer is applied to combine the multi-granularity clues and generate MCNN representations. The number of its hidden units $$ N $$, namely the dimension of representation, has effects on the combination of those patches. Changing the number of hidden units $$ N $$, we explore the relations between the dimension of MCNN representations and classification accuracy with well-aligned multi-granularity facial patches. The comparison of different dimensions is shown in Figure 10, which indicates that the number of dimensions almost has no influence on the convergent speed, but 256-dimensional representations achieve the highest accuracy. Therefore, it is reasonable for us to choose the number of hidden units as 256.

Figure 10. The comparison of different dimensional MCNN representations on accuracies and convergent performance achieved by CNN on the test set of the static image set in the daytime.

4.3.4 The significance of LSTMs

We first apply MCNN to detect driver drowsiness on videos, but it has no capacity to capture the temporal clues. To address this limitation, we consider MCNN+LSTMs. It is necessary to compare the situation with LSTMs ^[50] and without LSTMs to understand the effects of LSTMs. All experiments at this part are carried out on the FI-DDD dataset in daytime scenarios. The parameter settings and adjustments follow the settings in ^[51].

Parameters setting: The representations given by MCNN extractors are 256-dimensional, and the number of hidden units in each LSTM block is equal to 256. The forget gate is enabled, and the max memory step is set to 60 frames. We randomly select a batch with 1, 000 samples to train the LSTM parameters with a learning rate $$ 3e^{-4} $$. The fully connected layer projects the states of the last LSTM block to a 2-dimensional vector, which is decoded to the probability of drowsiness by a softmax operation.

MCNN-Only vs. MCNN + LSTMs: The experiments are carried out on four different granularities to research the effects of multi-granularity and LSTMs. Figure 11 shows the accuracy of MCNN only and MCNN + LSTMs for detecting videos on test sets under different granularities. The MCNN-Only method obtains 72.7% accuracy, while the approach of MCNN + LSTMs surpasses it by 15.6%. The reason is that the LSTMs have the ability to mine the clues in the temporal dimension, which is significant for recognizing lots of ambiguous states, such as closing eyes and blinking. Comparing the accuracies of different granularities, we discover that the well-aligned multi-granularity facial patches still achieve the best performance. The accuracy of the main parts ranks second, which means the granularity of the main parts certainly plays the most important role in improving the effectiveness compared to the other two granularities.

Figure 11. The comparison of accuracies achieved via MCNN-Only and MCNN + LSTMs on the test set of the FI-DDD dataset. Different granularities are still presented.

Comparisons with the previous methods

We evaluate the whole method on the evaluation set and compare it with the previous methods ^{[11, 13, 25, 52, 53]} achieved on the same dataset. Due to the long-term memory characteristics of the NTHU-DDD dataset, the max memory length is set to 120 frames, and other parameters remain the same as in the above experiments. Especially for night scenarios, we retrain a model with the night data of NTHU-DDD to detect driver drowsiness on near-infrared videos.

Accuracy:Table 3 presents the comparison of our method, the previous work ^{[11, 13, 25, 52, 53]}, and the proposed method achieves 90.05% accuracy, which is significantly improved compared to other existing methods, as the state-of-the-art method of driver drowsiness detection.

Speed:Table 3 shows the performance comparison of our method with other existing methods. The proposed method achieves a speed of 37 FPS on the GPU platform, satisfies the real-time performance requirements, and exceeds the majority of existing methods, second only to the methods proposed by Yu et al.^[52]. However, our method has significantly improved accuracy compared to their methods. At the same time, we measure the time consumption of all modules of our proposed method. From Table 4, CNN is the most time-consuming, and the approach achieves about 3 FPS on a CPU platform.

Table 4

Time consumption of each module of the proposed method. Others include reading, writing, and some converting operations

	Mg	CNN	LSTMs	Others	Total
CPU(E5) + GPU(M40)	11.1	10.7	0.6	4.5	26.9
CPU(I7)	5.6	302.3	0.6	3.2	311.7

Although our method has achieved good performance on existing datasets, there are still complex conditions and uncertain factors in real-world scenarios, such as significant changes in lighting and occlusion of the driver’s face. In future research, we will continue to explore the model explainability and uncertainty quantification^[54]. We will also consider applying the method proposed in this paper to real-world scenarios and continue to explore how to improve the generalization of the method under complex conditions.

Authors’ contributions

Made substantial contributions to the conception and design of the study: Liu T, Chen D, Yuan Z

Performed data analysis and interpretation: Zhang H, Lyu J

All authors were involved in the writing of the paper.

Availability of data and materials

Not applicable.

Financial support and sponsorship

This work was supported by the Beijing Natural Science Foundation (L201022).

Conflicts of interest

Not applicable.

Ethical approval and consent to participate

Written informed consent was obtained from the participants.

Consent for publication

Photographs of the faces used in this study were permitted and processed to protect privacy.

Copyright

© The Author(s) 2023.