​

Prof Richard Riley gives the following talk "Stability of Clinical Prediction Models Developed Using Statistical or Machine Learning Approaches" to raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks).

 

He defines four levels of model stability in estimated risks moving from the overall mean to the individual level. Then, through simulation and case studies of statistical and machine learning approaches, he show instability in a model’s estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data.