Selecting severity models

Obtaining a suitable loss severity distribution is a challenge for firms modelling operational risk. Marcelo Cruz assesses the goodness-of-fit tests that are used to validate severity distributions

Statistical distribution modelling in operational risk has made significant progress in the past 10 years. As is widely known, for firms using the loss distribution approach (LDA) the operational value-at-risk is calculated using a combination of frequency and severity. Frequency is usually modelled by the Poisson distribution, with many firms also using the negative binomial as an alternative. Thus, selecting a frequency distribution is hardly an issue for most firms. That is not to say that frequency distribution has little capital impact; it certainly does. One can verify this impact in situations where the loss collection threshold is reduced and an increase in operational risk capital is common in such situations.
However, in most cases loss severity is the determinant in the capital calculation. Selecting the proper severity distribution among the many options is not so easy. Fitting a single data set to different distributions might translate into completely disparate capital figures. Thus, justifying a severity distribution model can become a complex issue and often dominates discussions in regulatory examinations. So how do firms select their severity models?
There are a number of goodness-of-fit tests to validate severity distributions. The most popular type of test is a semi-parametric procedure that assesses if a hypothesised distribution fits the data. Usually this is referred to as an empirical distribution function (EDF) test. These tests can be divided into two classes: the supremum and quadratic. Some tests have a supremum and a quadratic version. The most popular test within the supremum class is the Kolmogorov-Smirnov (KS) test. On the quadratic side there are tests like the Anderson-Darling (AD) and the similar Cramer-von Mises (CVM) test. These EDF tests try to determine if two data sets differ significantly by comparing the data fitted to a particular (desired) distribution against an empirical distribution.
KS is non-parametric and distribution free. That is, it makes no assumption about the distribution of data so any distribution would work. More formally, the test is defined by the following formula Dn=sup|Fn(x) - F(x)|, with Fn(x) the desired distribution and F(x) the empirical distribution. Under the null hypothesis, the data follows the specified distribution, otherwise the test fails. The test is quite simple. First we have to calculate the cumulative distribution of the distribution. We proceed by doing the same to an empirical distribution (there are quite a few algorithms and formulas to do that, such as P= r--n+1 , where r is the ranking position and n is the number of data points).
Having calculated the cumulative distribution for the desired distribution and the empirical one, we need to find the greatest distance between the observed and expected cumulative frequencies, which is called the D-statistic. We compare this number with the critical D-statistic for that sample size. If the calculated D-statistic is greater than the critical one, we can reject the null hypothesis that the distribution is of the expected form.
To understand the general idea of these types of tests, I plotted a graph using 14 loss data points fitted to a lognormal distribution against the empirical distribution. It is easy to see two significant deviations from the empirical function, measured by D+ and D-. The next step is to check these values against the KS table and see if lognormal can be an acceptable distribution for this small data set.
Life would be too easy if simply by running this test we could reach a final answer on severity selection. The reality in operational risk is quite different. Because most firms use a loss collection threshold, distributions will be truncated. Also, operational loss databases usually have a number of extreme events and are quite dispersed. These characteristics make KS tests too lenient and basically useless to help a modeller select distributions as a stand-alone test.
The main concern of an operational risk modeller is not the body of the distribution but the tail. We need to test whether a distribution fits the data well mainly in the upper tail. The fit in the body is less important. A derivation of the AD test for the upper tail, noted ADup for the supremum case and AD2up for the quadratic case was introduced by Chernobai et al in Composite Goodness-of-Fit Tests for Left-Truncated Loss Samples (working paper, 2005). These tests are much more suitable to deal with the types of databases presented in operational risk and used by most firms to assess their fits.
All these tests bring some degree of comfort to the modeller. But the reality is that, due to the operational risk loss data characteristics and the sensitivity of modelling tails, the solution to determine the best fit distribution might come in the form of a scorecard that weighs the many tests and also adds an important component - the stability. That is to say, the preferred distribution would be the one that is already in use, so operational risk capital would not vary significantly.

Click here to read this column in pdf form