Journal of Risk Model Validation

Risk.net

How accurate is the accuracy ratio in credit risk model validation?

Marco van der Burgt

  • This paper introduces several methods to calculate the sample variance in the accuracy ratio and the area under the curve.
  • The first method is based on numerical integration and gives the best estimate of the sample variance;
  • The method, based on assuming normally distributed scores, and the method, which free from score distribution assumptions, give reasonable estimates of the sample variance;
  • The accuracy ratio and area under the curve are normally distributed

The receiver operating curve and the cumulative accuracy profile visualize the ability of a credit scoring model to distinguish defaulting from nondefaulting counterparties. These curves lead to performance metrics such as the accuracy ratio and the area under the curve. Since these performance metrics are sample properties, we cannot draw firm conclusions on the model performance without knowing the sampling distribution or the sample variance. We present four methods to estimate the sample variance of the accuracy ratio and the area under the curve. The first method is based on numerical integration, the second and third methods assume specific score distributions, and the fourth method uses a correlation, leading to a distribution-independent equation for the sample variance. We demonstrate by simulations that the first method gives the best estimation of the sample variance. The distribution-independent equation gives reasonable estimations of the sample variance, but ignores higher-order effects that are distribution dependent.

To continue reading...

You need to sign in to use this feature. If you don’t have a Risk.net account, please register for a trial.

Sign in
You are currently on corporate access.

To use this feature you will need an individual account. If you have one already please sign in.

Sign in.

Alternatively you can request an individual account here: