Journal of Risk Model Validation

Steve Satchell
Trinity College, University of Cambridge

This issue of The Journal of Risk Model Validation is made up of four papers. Readers worried that the “AI tsunami” might engulf them can relax: these papers are very much business as usual for the journal. AI is omnipresent – but as an enhancement to analysis, as it should be.

The issue’s first paper, “Generative artificial intelligence in model risk management: emerging opportunities, supervisory challenges and validation frameworks” by Arun Maheshwari, examines the intersection of generative AI (GenAI) and model risk management in relation to emerging regulatory expectations, model validation frameworks and practical applications. Drawing on empirical experience in model risk governance, Maheshwari proposes a structured approach to validating GenAI systems in line with the principles of US Federal Reserve Supervisory Letter SR 11-7, Prudential Regulation Authority Supervisory Statement SS 1/23 and Basel Committee on Banking Supervision AI risk management guidance. This paper also presents a real-world case study on validating an alert-reduction model for sanctions screening, based on a large language model. The difficulty in this area is that the extremely rapid development of AI means that regulation will always find itself trying to catch up. This paper is an excellent treatment of a challenging topic.

The second paper in the issue, “Graph neural networks for credit default prediction: robustness and model evaluation”, is by Konstantinos Papalamprou and Nikolaos Terzis. The perennial problems in credit default prediction revolve around imbalanced data: too few defaults, no ex post information about applications that have been previously rejected, and so forth. It is claimed that graph-based models can address these issues. Papalamprou and Terzis evaluate the robustness and performance of such graph-based models. Two inductive graph neural network (GNN) architectures – GraphSAGE and the graph attention network (GAT) – are implemented within a framework that integrates automated hyperparameter optimization, imbalance-aware loss functions and stress testing. Borrowers are represented as nodes in a k-nearest neighbor graph constructed from financial and demographic features. Model tuning is performed via Optuna, an optimization framework, while robustness is examined under the fast gradient sign method and projected gradient descent perturbations, with adversarial training enhancing stability. The authors’ results indicate that optimized GNNs trained in the adversarial manner outperform classical baselines in area under the curve and F1 metrics; the authors also claim that these procedures show resilience under feature perturbations. These findings highlight the importance of robustness evaluation as part of the broader model assessment process for modern credit risk modeling.

The editorial board and I always welcome new research on backtesting. It is fair to say that nobody really knows what interest rates will be in five years’ time, and modelers address this by creating scenarios. However, doing this creates a range of validation issues. This issue is examined by Krishan Kumar Sharma in our third paper, “A dual backtesting framework for quantifying nested model error and unlocking capital efficiency”. Macroeconomic scenario forecasting serves as a cornerstone of regulatory capital exercises such as Comprehensive Capital Analysis and Review (CCAR), Current Expected Credit Loss (CECL), International Financial Reporting Standard 9 (IFRS 9) and the Internal Capital Adequacy Assessment Process (ICAAP), and it introduces a critical and frequently overlooked problem: nested errors. These errors originate from third-party scenarios and the internal modeling process, and they compound nonlinearly, as they are assumed within preprovision net revenue (PPNR) and credit loss forecasting models. This compound result can lead to distorted capital projections, highlighting the need for a robust solution. To address these issues, Sharma proposes a dual backtesting framework: single-blind backtesting evaluates core models, while double-blind backtesting assesses the entire system. The difference between the two tests can be used to quantify the error from the macro scenario forecast in PPNR and credit loss forecasting models. This methodology provides an interesting approach, which adheres to the “use test” principle in SR 11-7, facilitating a more insightful comprehension of its application.

The issue’s final paper, “A novel budget-based CCSVM model for credit risk prediction” by Zhe Li, Yuxiao Jia,Weijiang Ma, Zhimin Zhao and XushiWei, starts with the assertion that credit risk prediction identifies default risk by establishing a nonlinear relationship between feature data (explanatory variables) and default status. The feature system is the crucial factor affecting the accuracy of credit risk prediction. This paper addresses how to identify customers within a certain range of cost budgets. A case study on farmers in China shows that the characteristics of incomplete loan information, dispersed residences and strong liquidity indicate that bank lending should consider not only the identification of default customers but also the actual cost budget. A modified support vector machine (CCSVM), which considers the cost of obtaining credit information and the budgets of decision makers, is established to optimize the credit evaluation results. Choosing the objective function to be optimized for any risk problem is a matter of judgment. Li et al perform feature selection and weighting by optimizing the minimum total misjudgement cost as the objective function while constraining the prediction deviation, feature weights and cost budget. Their empirical results for a sample of Chinese farmers demonstrate that the proposed CCSVM model is effective in selecting credit information costs, and that decision makers can achieve the necessary prediction accuracy with lower costs. To my mind, explicitly including the costs in assessing default, along with the direct costs of default, seems highly sensible.