Validation of deep learning models for cuffless blood pressure estimation on a large benchmarking dataset

INTRODUCTION

Cuffless blood pressure (BP) monitoring has great value in cardiovascular healthcare. Unlike traditional cuff-based methods, which can cause discomfort and are impractical for long-term monitoring, cuffless technology allows for continuous monitoring of BP changes without interrupting daily activities^[1]. This continuous monitoring is particularly valuable for patients with hypertension and other cardiovascular conditions by providing a comprehensive picture of BP fluctuations in various situations^[2], thus facilitating early detection and timely intervention. Moreover, cuffless BP technology also empowers individuals to be aware of their BP variations and proactively manage their health^[3].

A plethora of machine and deep learning models for cuffless BP estimation have been proposed in recent years^[4-9]; however, comparison of their performance is challenging as different models have been validated on diverse datasets. These models can be categorized into three types: subject-specific, population-based, and hybrid ones^[1]. Subject-specific models, although accurate, require frequent calibrations and often rely on data from individuals, which may be impractical^[4,10,11]. Furthermore, the lack of reported intra-subject BP variations in these studies raises concerns about potentially over-optimistic results. In contrast, population-based models, which do not need calibration, often sacrifice accuracy due to the complex relationship of the features and BP across diverse populations^[12,13]. These models must consider basic demographic information in their design. The hybrid method seeks a balance between practicality and accuracy but has been limited by small datasets in previous validations^[12-14].

Emerging studies on cuffless BP estimation have validated various models on large-scale datasets. One study, using the VitalDB dataset, validated a pulse arrival time-based BP estimation model. They reported very satisfactory results^[15], which may be attributed to their focus solely on the accuracy of linear regression, without validation on an independent dataset. Another study proposed a lightweight deep learning model, KD-informer, and accurately predicted continuous BP on two separate datasets, but the intra-subject BP variations were not reported^[8]. Additionally, a recent study assessed several machine learning models such as ridge regression, SVM, AdaBoost, and random forest, as well as deep learning methods including VGGNet16, ResNet50, BiLSTM, and ResLSTM, on a large-scale dataset gathered from 3,077 individuals without severe cardiovascular diseases using smartwatches^[16]. They reported that the best-performing calibration-free model had estimation errors of - 0.71 ± 13.04 and - 0.29 ± 8.78 mmHg for SBP and DBP, respectively. However, the BP range of this literature was limited and they only tested under static conditions. Furthermore, their study highlighted significant performance degradation in models for aged (≥ 55 years old) and hypertensive subgroups. One of our previous works proposed a high-performance deep learning model, UTransBPNet^[17]. Due to its strong capability in short- and long-range feature representation due to the combined structure of Unet and Transformer, it demonstrated superior performance over popular models such as CNN-LSTM-attention^[17] and CNN-BiGRU^[18], in scenarios with significant intra-subject BP variations. However, this validation was limited to a small, self-collected dataset with a narrow demographic range. The real-world effectiveness of UTransBPNet, particularly in clinical settings with diverse demographics and sufficient intra-subject BP variations, remains unknown.

Addressing this gap, Wang et al. recently proposed a large, cleaned dataset PulseDB with diverse demographics and sufficient intra-subject BP variations for benchmarking cuffless BP estimation methods^[19]. Leveraging this resource, our study aims to validate the performance of several deep learning models in estimating arterial BP (ABP) across a diverse population in intensive care units (ICUs) and with sufficient intra-subject BP variations. We further explore three approaches to optimize the performance of UTransBPNet, incorporating demographic information and employing a hybrid methodology.

METHODS

Data

This study utilized the PulseDB dataset, which originates from the large-scale MIMIC-III and VitalDB datasets^[19], and includes several continuous physiological signals like electrocardiogram (ECG), photoplethysmogram (PPG), and arterial BP (ABP) acquired from ICU patients. To investigate the impact of demographic information on BP estimation, only the VitalDB dataset was used, as the MIMIC-III dataset did not include this information. A series of data preprocessing steps were conducted on the dataset as below:

Band-pass filtering and resampling. ECG, PPG, and ABP signals were filtered at 0.5-30 Hz, 0.5-15 Hz, and 0.5-15 Hz, respectively, and then resampled to 125 Hz.

Normalization. Maximum-minimum normalization was applied within each segment for ECG and PPG separately to scale the values to the range between 0 and 1.

Removing segments with abnormal BP. The segments with ABP values beyond the normal physiological range between 40 and 250 mmHg were excluded.

Removing segments with abnormal ECG amplitudes. The amplitude differences of adjacent ECG peaks within one segment were calculated, and those with a difference ≤ 0.5 were excluded.

Removing segments with abnormal heart rate. The segments with the R-R interval beyond the range of 0.4 to 2.4 seconds were removed.

Excluding patients with short segment lengths. The patients with a total duration of signal recording < 20 min were excluded.

With preprocessing by these steps, a total of 1,257,141 10-sec signal segments of 2,154 patients finally remained in the following analysis. The demographic characteristics of these patients are presented in Table 1. The dataset spans a diverse age range from 18 to 97 years old. Figure 1 shows the histograms for systolic BP (SBP), diastolic BP (DBP), mean BP (MBP), the intra-subject max-min SBP and DBP variations (denoted as ΔSBP_max-min and ΔDBP_max-min, respectively) of the entire dataset. The dataset presents substantial intra-subject max-min SBP and DBP variations, ranging from 20 to 180 mmHg (mean ± SD: 74.2 ± 26.3) for SBP, and 20 to 180 mmHg (43.6 ± 20.5 mmHg) for DBP, respectively.

Figure 1. (A-C) Distributions of SBP, DBP, and MBP; (D-F) distributions of the subject age, and intra-subject max-min variations of SBP and DBP; The y-axis of (A-C) represents the number of 10-second signal segments, while the y-axis of (D-F) indicates the number of subjects across the entire dataset.

Table 1

Demographic and cardiovascular characteristics of the subjects included

Characteristics	Values
Num of subject	2,154
Age, years old 18-27 28-37 38-47 48-57 58-67 68-77 78-87 88-97	Number (Proportion, %) 61 (2.83%) 120 (5.57%) 229 (10.63%) 441 (20.47%) 602 (27.95%) 495 (22.98%) 190 (8.82%) 16 (0.74%)
Female, %	55.2%
Male, %	44.8%
Height, cm (mean ± SD)	162.72 ± 8.70
Weight, kg (mean ± SD)	61.34 ± 11.42
BMI, kg/m2 (mean ± SD)	23.09 ± 3.46

Model

Two model structures proposed in our previous study were evaluated on this large dataset: one is the CNN and bidirectional LSTM network with attention mechanism (CNN-LSTM-ATT) and the other is UTransBPNet^[17]. UTransBPNet comprises three primary modules: an enhanced U-net for capturing short-range features of the input signals, a transformer module for obtaining long-range contextual information, and a cross-attention module to further refine feature learning. Based on UTransBPNet, three methods were further introduced, as shown in Figure 2:

Figure 2. Model structure. (A) UTransBPNet; (B) UTransBPNet_Demo_In; (C) UTransBPNet_Demo_Last; (D) UTransBPNet_Cal.

UTransBPNet model: it uses ECG, PPG, and the first and second derivatives of PPG signals (VPPG and APPG) as inputs to estimate continuous ABP.

UTransBPNet model with individual calibration (UTransBPNet_Cal): similar to UTransBPNet, UTransBPNet_Cal incorporates calibration using the initial 20 seconds of individual data. Specifically, the BP bias that minimizes the difference between the reference and predicted BP during the initial 20 seconds was identified and used to calibrate the entire set of predictions for each subject.

UTransBPNet model with demographic inputs (UTransBPNet_Demo_In): it integrates demographic information along with the ECG and PPG as inputs. Each demographic feature was duplicated to match the length of the input signal. The pure feature values were used for the continuous variables such as age, height, weight, and BMI, while binary coding was used for discrete variables such as gender.

UTransBPNet fused with demographics (UTransBPNet_Demo_Last): the demographic features were directly concatenated with the deep features learned by the model at the last fully-connected layer of UTransBPNet.

Training & validation. The training and validation of all models were conducted on Pytorch using an Nvidia Tesla V100 graphics card with 32 GB of memory. During the training of each model, the hyperparameters were configured as follows. The learning rate was set as 1e-5 with a batch size of 64 to balance the trade-off between computational efficiency and the accuracy of gradient estimates. The training was scheduled to run for 100 epochs, with an early stopping mechanism activated after 10 epochs without improvement to prevent overfitting. The L1 Loss function and the Adam optimizer were employed. A weight decay parameter of 0.1 was employed to regularize the model. Ten-fold subject-independent cross-validation was performed. In each fold, the training and testing datasets did not include data from the same subject.

RESULTS

Table 2 and Table 3 show the predictive errors and Pearson’s correlation between the reference and predicted BP for the five models, respectively. As seen in Table 2, UTransBPNet achieved the lowest MAEs, significantly outperforming CNN-LSTM-ATT, UTransBPNet_Cal, and UTransBPNet_Demo_Last. UTransBPNet_Demo_In achieved comparable performance with UTransBPNet for ABP, DBP and MBP, except for an increase of around 0.5 mmHg of MAE for SBP prediction (P = 0.0073). From Table 3, among all models, UTransBPNet_Demo_In demonstrated the highest correlation between the reference and predicted BP at both individual and population levels, highlighting the importance of integrating demographic information as input data. All models presented notable individual differences in performance, as reflected by the SDs of MAE_ind and r_ind. Figure 3 provides a visualization of the estimation results of the four UTransBPNet models for two representative individuals. For one subject [Figure 3A], UTransBPNet_Demo_In demonstrated the most accurate tracking of the reference ABP, while all models failed to track the reference accurately in the other subject [Figure 3B].

Figure 3. Typical examples of ABP predictions given by the four models.

Table 4 presents the correlation and significant analysis results between the intra-subject MAEs of ABP predictions and relevant influencing factors including demographic and cardiovascular characteristics. There is no significant correlation between the demographic features and the MAEs for all four models, suggesting that the performance of the four models does not vary with demographic factors. The performance across different age groups for the four models is further presented in Figure 4. Among all models, UTransBPNet_Demo_In presented the most robust performance among different age groups, exhibiting no significant differences between groups. In contrast, the other models presented one or more significant differences between groups. The scatter plots and the fitted linear relationship between the demographics and the MAEs given by UTransBPNet_Demo_In, as presented in Figure 5, further highlight UTransBPNet_Demo_In’s superior ability to maintain consistent accuracy irrespective of age, height, and weight.

Figure 4. Analysis of the model performance across various age groups, with error bars representing the SDs of MAEs for each group. Significant testing was performed among age groups for each of the four models. *: the predicted Mean Absolute Errors (MAEs) for the last age group are significantly different from those of the other groups across each model, achieving statistical significance with a P-value of less than 0.01 (P < 0.01).

Figure 5. Scatter plots and insignificant linear relations between the demographics and the MAEs of UTransBPNet_Demo_In.

Figure 6. Scatter plots and linear relations between the intra-subject cardiovascular features and MAEs of the three models.

On the other hand, all models except UTransBPNet_Demo_In presented significant correlations between the MAEs and the cardiovascular features, indicating that the model performance was influenced by cardiovascular features (defined as significant when r > 0.15 and P < 0.05). Specifically, the UTransBPNet model shows a notable positive correlation between the MAE and two cardiovascular features: mean and SD of SBP (denoted as SBP_mean and SBP_std). This indicates relatively poorer performance in individuals with high SBP levels and fluctuations. In contrast, the UTransBPNet_Demo_Last model demonstrates poorer performance primarily in individuals with faster heart rates. The UTransBPNet_Cal model tends to be influenced by most factors, performing less effectively in individuals with significant SBP and DBP fluctuations, high maximum SBP and DBP, broad SBP and DBP ranges, as well as high heart rate variations. The scatter plots and fitted linear relationship graphs in Figure 6 further highlight these trends. Out of all models, the UTransBPNet_Demo_In model stands out with the most stable performance, exhibiting robust predictive performance unaffected by individual cardiovascular features.

DISCUSSION

While numerous studies have developed new models for cuffless BP estimation, the validation was insufficient in terms of the demographic diversity of subjects and the intra-subject variations. This study investigated the effectiveness of four UTransBPNet-based methods and a CNN-LSTM-ATT model for ABP estimation in a large ICU population characterized by diverse demographics and cardiovascular conditions. Among these methods, the UTransBPNet_Demo_In model, which integrates demographic information with ECG and PPG as inputs, demonstrated superior performance in tracking BP variations. Although UTransBPNet_Demo_In presented notable individual variations in performance, it demonstrates robust accuracy across different demographics and cardiovascular conditions. The superior performance of UTransBPNet_Demo_In may be due to the capability of learning effective features representing the complex interplay between demographics and cardiovascular signals through the deep learning process, which ensures robust BP estimation performance across diverse ICU demographics and cardiovascular states. In contrast, the UTransBPNet_Demo_Last method, which directly merges demographic data with deep representations of physiological signals without an extensive deep learning process, fails to effectively learn relevant features. Additionally, the UTransBPNet_Cal method, which utilized a bias calibration technique, exhibited reduced performance. This decline is likely due to the unrealistic assumption that vascular states remain constant over long periods.

For monitoring patients in the ICU, continuous BP waveform is invaluable as it offers detailed insights into the patient's hemodynamic status. Therefore, we have reported the accuracy of ABP to cater to this critical requirement. However, we also recognize that other clinical settings may prioritize discrete measurements of SBP, DBP, and MBP over continuous waveforms, and many previous studies have only reported the prediction performance of models for SBP and DBP^[8,16]. To facilitate a broader comparison with existing research, we have included performance metrics for both SBP and DBP predictions in this study.

This study has several limitations. Firstly, there is a noticeable gap between the performance of the models and the international standards for BP measurement. This discrepancy may be attributed to the complexity of the PulseDB dataset, characterized by highly dynamic BP variations in ICU settings. Additionally, the data preprocessing methods employed, such as normalization within segments, might neglect useful temporal information in physiological signals. Another significant limitation is the relatively intuitive approach used to integrate demographic features into the deep learning model. Future research should focus on developing more effective and dedicated methods for integrating demographic data and deep features extracted from continuous physiological signals. Moreover, a statistical analysis of the data distribution for the entire dataset after excluding certain segments found that, the distribution did not align with the data distribution requirements set by the AAMI standard, notably with less than 5% of the measurements falling into the very high categories of BP (> 160 and 100 mmHg for SBP and DBP, respectively). Therefore, further validation that strictly adheres to the AAMI test standard is needed in future work.

CONCLUSION

In conclusion, this study comprehensively validated two deep learning models for cuffless BP estimation on a large-scale ICU dataset, and designed different approaches to enhance the generalization capability across diverse demographic and cardiovascular conditions. Among these, the UTransBPNet_Demo_In model, leveraging the integration of demographic data with ECG and PPG signals as model inputs, has emerged as the most effective one. Although there is still considerable room for improvement due to the challenging nature of cuffless BP estimation, especially within the complex context of ICU settings, this work provides a benchmark for future studies. Future advancement may focus on enhancing model accuracy by more effectively leveraging the temporal trends of physiological signals.