Uncertainty estimation affects predictor selection and its calibration improves materials optimization

Uncertainty is crucial when the available data for building a predictor is insufficient, which is ubiquitous in machine learning driven materials studies. However, the impact of uncertainty estimation on predictor selection and materials optimization remains incompletely understood. Here, we demonstrate that in active learning, uncertainty estimation significantly influences predictor selection, as well as that the calibration of uncertainty estimation can improve the optimization. The idea is validated in three alloys datasets where three commonly used algorithms are used to build predictors with similar accuracy. It is shown that the XGBoost (extreme gradient boosting) presents more reliable uncertainty estimation than SVR (support vector regression) and NN (neural network). Using the directly estimated uncertainty for the three predictors with similar accuracy, we find that the optimization is quite different, which suggests that uncertainty estimation plays a role in predictor selection. The uncertainty estimation is then calibrated to improve the reliability and its effect on optimization is compared to the uncalibrated one. In the 9 cases (3 models and 3 datasets) we considered, 8 show improved optimization with calibrated uncertainty estimation. This work suggests that uncertainty estimation and its calibration deserve more attention in active learning-driven materials discovery.