Model risk quantification for machine learning models in credit risk

Lukasz Prorokowski

Abstract

The assessment of model risk is the responsibility of the validation function. This validation requirement was recently underscored by the European Banking Authority’s “Supervisory handbook on the validation of rating systems under the internal ratings based approach” (EBA/REP/2023/29). Further, Capital Requirements Regulation III, applicable from January 1, 2025, introduces a detailed definition of model risk, which provides the principles to which the model risk assessment should adhere. On the other hand, the rapid surge in data mining capabilities coupled with technology-enabled innovation in credit risk provides an opportunity for utilizing machine learning models (MLMs) instead of common internal ratings-based (IRB) models. Against this backdrop, the regulatory consultation conducted by the European Banking Authority in 2023 (EBA/REP/2023/28) suggests that banks intend to use MLMs in selected areas of the IRB approach. This paper reviews bank-specific model risk measurement techniques with a focus on implemented model risk rating solutions for MLMs, and it reports the key challenges faced by the validation functions at participating banks in relation to quantifying MLMs’ model risk. As it transpires, all of the participating banks face challenges with implementing an effective model risk rating tool for the commonly used IRB model, as well as for more complex quantitative constructs such as MLMs. Recognizing these challenges, the paper delivers a model risk rating concept for MLMs in a dual form, for both the model life cycle and the MLMs already in production. The proposed solution uses decision trees to eliminate the bias in the model risk assessment based on expert judgment by the internal validation member.