Comparison of strategies for validating binary logistic regression models

Both systems SAPS II and SAPS 3 calibrated poorly in this study.

SAPS II did not show significant deviations from ideal calibration in the first two deciles of risk, but it overpredicted mortality in higher-risk classes.

The AU-ROC ranges from 0.50 (no discrimination: complete binary random of 50 % similar to flipping a coin) to 1.00 (100 % correct discrimination of the model) [].Calibration along with discrimination is an important measure of accuracy to validate predictive logistic regression models.Most predictive models in intensive care such as Simplified Acute Physiology Score (SAPS) II [] consider the binary outcome whether a patient will be alive or dead at hospital discharge.SAPS 3 overpredicted mortality homogeneously across all risk classes and showed a higher overprediction rate than SAPS II.Similarly, a common finding in most external validation studies is that models show poor calibration, while discrimination is usually good for all of them [].On the basis of the results of Poole et al., changes over time may be a possible interpretation as to why the older score is not considered better than the more recent one.In this way, continuous improvements in health care possibly have a bigger impact on sicker patients, whereas unchanged conventional treatments have been given to low-risk patients since a long time ago.The H–L test is easy to compute and its interpretation is intuitive, but it has acknowledged limitations such as being very sensitive to sample size [].The traditional plot or calibration curve also has some disadvantages: first, rather than a curve, it is a jagged line connecting the points in the plot; second, it is not accompanied by any information on the statistical significance of deviations from the bisector [] present a multicenter study aimed at comparing the performance of SAPS II and SAPS 3 in predicting hospital mortality.If the model calibrates well, there will not be a substantial deviation from the 45° line of perfect fit or bisector.On the contrary, miscalibration of the model will be a function of expected probability.

Leave a Reply

Your email address will not be published. Required fields are marked *

One thought on “comparison of strategies for validating binary logistic regression models”