A dual backtesting framework for quantifying nested model error and unlocking capital efficiency

Krishan Kumar Sharma

Abstract

Macroeconomic scenario forecasting serves as a cornerstone for regulatory capital exercises such as Comprehensive Capital Analysis and Review (CCAR), Current Expected Credit Loss (CECL), International Financial Reporting Standard 9 (IFRS 9) and the Internal Capital Adequacy Assessment Process (ICAAP), and it introduces a critical and frequently overlooked problem: nested errors. These errors originate from third-party scenarios and the internal modeling process, and they compound nonlinearly as they move through pre-provision net revenue and credit loss forecasting models. This compound result yields distorted capital projections that can lead to either wasteful overcapitalization or dangerous capital shortfalls, highlighting the need for a robust solution. To address these issues, this paper proposes a dual backtesting framework: single-blind backtesting evaluates core models, while double-blind backtesting assesses the entire system. The difference between the two tests can be used to quantify the error from the macro scenario forecast in pre-provision net revenue and credit loss forecasting models. This methodology provides a comprehensive solution adhering to the “use test” principle in US Federal Reserve Board Supervision and Regulation Letter 11-7, facilitating a more profound comprehension of its application. Through the precise quantification of these nested errors, financial institutions can make efforts to minimize errors driven by upstream macro components, enabling them to transition from a strategic approach of blunt conservatism to a more nuanced and accurate calibration, thereby underscoring the strategic significance of this methodological shift. This transition should unlock significant capital efficiency, enhance profitability and build stronger regulatory trust. Our novel framework establishes a new standard in model risk management by proving robust validation and established guidelines for improving accuracy, providing a decisive strategic advantage.