Journal of Risk Model Validation

The effect of imperfect data on default prediction validation tests

Heather Russell, Douglas Dwyer and Qing Kang Tang


Analysts often find themselves working with less than perfect development and/or validation samples, and data issues typically affect the interpretation of default prediction validation tests. Discriminatory power and calibration of default probabilities are two key aspects of validating default probability models. This paper considers how data issues affect three important power tests: the accuracy ratio, the Kolmogorov-Smirnov test and the conditional information entropy ratio. The effect of data issues upon the Hosmer-Lemeshow test, a default probability calibration test, is also considered. A simulation approach is employed that allows the impact of data issues on model performance, when the exact nature of the data issue is known, to be assessed. We obtain several results from the tests of discriminatory power. For example, we find that random missing defaults have little impact on model power, while false defaults have a large impact on power. As with other common level calibration test statistics, the Hosmer-Lemeshow test statistic simply indicates to what degree the level calibration passes or fails. We find that the presence of any data issue tends to cause this test to fail, and, thus, we introduce additional statistics to describe how realized default probabilities differ from those expected. In particular, we introduce statistics to compare overall default probability level with the realized default rate, and to compare the sensitivity of the default rate to changes in the predicted default probability.

Only users who have a paid subscription or are part of a corporate subscription are able to print or copy content.

To access these options, along with all other subscription benefits, please contact or view our subscription options here:

You are currently unable to copy this content. Please contact to find out more.

You need to sign in to use this feature. If you don’t have a account, please register for a trial.

Sign in
You are currently on corporate access.

To use this feature you will need an individual account. If you have one already please sign in.

Sign in.

Alternatively you can request an individual account here