Journal of Risk Model Validation

Risk.net

Goodness-of-fit for discrete-choice models of borrower default

Arden Hall

  • For account level default models, rank-order statistics are insufficient to identify serious misspecification.
  • Misspecification can lead to inaccurate loss estimates even if the model is only used for portfolio segmentation.
  • Comparing predicted probability to actual frequency by segment can identify misspecification.  
  • Two format tests, Hosmer-Lemeshow and Standardized Pearson, can be effective when applied to samples.

Discrete choice models of the probability of default (PD) have several applications in finance. In some applications, such as credit scoring, their value is in ranking applicants or customers by PD. Other applications, such as estimating losses as part of determining capital requirements under Basel, require accuracy for the estimated PD. There is a well-developed set of tests to assess models’ ability to rank-order, and these are sometimes relied upon to assess the accuracy of models’ probability estimates. This paper demonstrates that the rank-order tests are unreliable for assessing models to be used to predict probabilities. This is true even when estimated probabilities will only be used to assign observations to segments. There are other tests, such as the Hosmer–Lemeshow test, which assess magnitude as well as order. While there are some practical difficulties in applying these alternative tests to the data sets typically used for default model estimation, they can provide a better assessment of how well a model fits the data.