Journal of Risk

Welcome to Volume 9 Issue 2 of The Journal of Risk. This issue is made up of 4 technical papers: The literature on backtesting financial risk models has long been dominated by two rival paradigms. The first is the "exceedance framework": this focuses on testing the frequency of exceedances over a particular value-at-risk (VAR) threshold. The best-known examples of these backtests are those of Kupiec (1995) and Christoffersen (1998), but there has also been a lot of good recent work within this framework, most notably the martingale-based approaches to backtesting proposed by Engle and Manganelli (2004) and others. Working within this framework allows us to apply some simple univariate tests, but the cost of this tractability is that exceedance-based approaches are potentially inefficient because focusing on exceedances over a single threshold means throwing away a lot of potentially useful information about exceedances over other thresholds.

An exceedance approach to backtesting can also be regarded as placing an exclusive focus on backtesting forecasts of a single quantile. It is then natural to ask if backtesting can be improved by testing the performance of multiple quantile forecasts at the same time, and this takes us to the rival "multiple quantile" framework. Examples of backtests of this type are those of Crnkovic and Drachman (1996), Diebold, Gunther and Tay (1998) and Berkowitz (2001). In the limit, as the last two studies exemplify, we might have a "multiple quantile" approach that backtests all quantile forecasts simultaneously. The principle advantage of working with "multiple quantile" procedures is the prospect of more powerful tests. The downside, in some cases, is that testing can become more difficult: for example, we can no longer use binomial tests. However, testing is not necessarily difficult: as Berkowitz (2001) shows, it is sometimes possible to apply simple tests of standard normality when working within a "multiple quantile" framework.

The articles in this special issue reflect a wide range of topical concerns in backtesting. The special issue leads with Sean Campbell's article A review of backtesting and backtesting procedures. This article provides an overview of different backtests, which are reviewed from both statistical and risk management perspectives. The article starts within the exceedance backtesting tradition and focuses on the properties of the unconditional coverage and independence of exceedances. It discusses how different backtest procedures can be classified in terms of whether they test for unconditional coverage or independence, or both. The article then addresses the "multiple quantile" issue and examines how the power of test procedures might be improved by testing VAR forecasts at multiple confidence levels. This issue is examined via a simulation study which examines the forecast performance of a variety of alternative VAR models. The results suggest that a "multiple quantile" approach to backtesting has the potential to deliver moderate improvements in power relative to an exceedance approach. They also suggest that the greater the misspecification, the greater the improvements in power. This is a helpful result for risk managers as it suggests that the greater their doubts about the model, the more important it might be to backtest quantiles at more than one confidence level.

The second article, Backtesting value-at-risk accuracy: a simple new test, by Christophe Hurlin and Sessi Tokpavi, proposes a new statistical backtesting procedure. This article reviews the limitations of the Christoffersen (1998) Markov chain approach, and then examines the more recently developed martingale approaches to backtesting of Engle and Manganelli (2004) and Berkowitz, Christoffersen and Pelletier (2005). These latter approaches have a number of attractions but are limited by the constraints of the exceedance framework, ie, they can only backtest quantile forecasts at a single confidence level. Hurlin and Tokpavi propose to overcome this limitation by working with a multivariate generalization of the Berkowitz et al. (2005)approach, based on the multivariate portmanteau test of Li and McLeod (1981). The Hurlin-Tokpavi approach allows us to apply martingale methods in a "multiple quantile" backtesting framework, and so combines the virtues of both. The article then compares the performance of multivariate and univariate martingale tests in a simulation exercise, and the results suggest that the multivariate tests have appreciably greater power, a result that is reminiscent of Campbell's earlier finding.

The next article, Quality control of risk measures: backtesting VAR models by Victor de la Pena, Ricardo Rivera and Jesus Ruiz-Mata, also proposes a new backtesting procedure, but one of a very different kind. Working within an exceedance framework, this article proposes a statistical backtesting procedure called quality control of risk measures (QCRM). The starting point for this procedure is to reverse the usual null and alternative hypotheses. Instead of working with the null hypothesis that the forecasting model is adequate, it suggests working with the null hypothesis that the model is not adequate, ie, we should presume that the model is inadequate unless there is strong evidence in its favour. The attraction of this approach is that it allows us to control the probability of wrongly accepting an incorrect model, something that cannot easily be done if we adopt the usual null hypothesis. QCRM also involves a pivoting
technique to obtain one-sided confidence intervals for the probability of an exceedance. The article goes on to compare the type I and type II error rates of the Basel and QCRM backtest procedures, and results suggest that QCRM delivers a less extreme trade-off which produces significant increases in power relative to the Basel approach.

The fourth article is Backtesting risk methodologies from one day to one year by Gilles Zumbach. This article proposes a new backtesting framework that is particularly suited to comparing different models and to backtesting models over relatively long forecast horizons. This framework uses forecast quantiles and exceedance differences to compute a variety of evaluation measures. The paper also proposes a new risk forecasting methodology called RM2006 and compares the performance of this approach with that of more traditional historical-simulation, equal-weight and RiskMetrics moving-average forecast approaches. The performance of this new approach is examined in a systematic study using 233 time series covering all geographic areas and asset classes. Results indicate that this approach correctly captures both the long memory of volatility dynamics and the heavy tails of the residuals, and that it systematically outperforms the older approaches at all forecast horizons.

The final article is Backtesting market risk models in a standard normality framework by Kevin Dowd. Unlike the other articles, it does not propose any new test or testing procedure. Instead, it seeks to extend the range of applicability of the backtesting strategy proposed by Berkowitz (2001), the gist of which is to transform our data so that they become iid standard normal under the null hypothesis of model adequacy and then hit them with a battery of tests of iid standard normality. This backtesting strategy was originally proposed in a rather limited context, which ignores many of the complications that often arise with real-world backtesting problems - such as the complications that arise with multiple-step ahead density forecasting models, multivariate models, multiple models at different business-unit levels and parameter uncertainty. The Dowd article examines ways in which this strategy can be successfully applied to problems involving these features, and suggests that the Berkowitz strategy is more powerful and more useful than has hitherto been appreciated. In so doing, it also offers some guidance on such issues as how to backtest multiple-step ahead risk models, how to backtest copula
models and how to carry out backtests in the presence of parameter uncertainty.

This special issue is very much a collective effort, and I am deeply grateful to The Journal of Risk Editor-in-Chief, Barry Schachter, for his guidance in putting this issue together. I would also like to thank the other authors for contributing such a fine set of articles and for their cooperation and assistance, and to thank the referees involved. I very much hope that the readers of the journal will find some of these articles helpful in their own work: there is no doubt that a great deal remains to be done.