Journal of Risk Model Validation

Quantifying model selection risk in macroeconomic sensitivity models

Joseph L. Breeden and Nikolay Dobrinov

  • Forecast accuracy is compared from lists of candidate models for a range of loan products.
  • Risks are also compared for ensembles of models.
  • Monte Carlo error propagation was found to accurately quantify model risk.
  • Error propagation showed that a single model’s forecasts can be systematically biased relative to the full distribution.

Model selection is a critical step in model development and brings with it a significant level of model risk. This analysis considers a real-world example and compares the forecasts and uncertainties of three possibilities: the model selected as best, the best ensemble and the model not selected. A key part of this analysis is propagating the prediction intervals for the individual forecasts using a Monte Carlo simulation all the way through the lifetime probability of default calculations to obtain distributions for the lifetime forecasts. The results highlight two sources of risk: a systematic difference between the expectation value of the forecast and the median of the simulated distribution, and the 95% confidence interval from the distribution. The difference between the forecast and the simulated median should be included as part of the loss reserve. The confidence interval from the distribution should contribute to the model risk component of the economic capital. This paper also illustrates how uncertainties can be computed for Current Expected Credit Losses or International Financial Reporting Standard 9 Stage 2 lifetime loss estimates.

Sorry, our subscription options are not loading right now

Please try again later. Get in touch with our customer services team if this issue persists.

New to View our subscription options

If you already have an account, please sign in here.

You need to sign in to use this feature. If you don’t have a account, please register for a trial.

Sign in
You are currently on corporate access.

To use this feature you will need an individual account. If you have one already please sign in.

Sign in.

Alternatively you can request an individual account here