US debate grows over value of stress testing

US law now mandates annual stress tests for banks - but their relevance to op risk managers is disputed

Is operational risk stress testing a fool’s errand or the future of the discipline? In the wake of the US Federal Reserve’s stress-testing programme results earlier this year, the debate on stress testing is heating up. Quants at the Richmond Federal Reserve Bank say they can demonstrate macroeconomic correlations with operational risk events, while some in the industry say such correlations are imaginary – or, at least, complex and potentially misleading. Other industry experts agree operational risk can be stress tested, and argue that it is only when you think a bit outside the box about how operational risk is correlated with the macroeconomic picture that the truth becomes clearer.

Nineteen US banks were required to participate in a stress-test programme, the Comprehensive Capital Analysis and Review (CCAR). Banks were asked to test the effect of various macroeconomic scenarios against their own operational risk loss profile. These scenarios focused on economic events, such as a contraction in real GDP, a US equity price collapse and a US house price slide. Recessions in Europe and Asia were also included.

Although the Fed had conducted stress-testing exercises during the financial crisis, this new, annual operational risk stress test programme grew out of a provision in the Dodd-Frank Act that requires the Federal Reserve to stress test non-bank financial companies as well as bank holding companies with more than $50 billion in assets annually. Firms with $10 billion in assets or more must also perform annual stress tests, but under a slightly less rigorous regime.

The large banks now must submit worksheets quarterly that contain their projections of operational risk losses, and information on how firms undertake the translation of their loss history into projected losses. For the first time, firms have had to submit all the historic loss data up to 2010 to the Federal Reserve, grouped by the seven event types used under the Basel II framework: internal fraud; external fraud; employment practices and workplace safety; clients, products and business practices; damage to physical assets; business disruption and system failure; and execution, delivery and process management.

Firms were asked to submit their own stress-test results, and the Federal Reserve then used the historic loss event data with its own set of scenarios for operational risk for the following nine quarters.

“Big operational risk losses – tail events – are not a reflection of a drop in business activity. They are a reflection of prior exuberance.” As a result, “stress testing does work if you take into account the historical period”

The results, released in mid-March, showed that collectively the 19 banks would suffer $534 billion in credit and market risk losses. Losses related to operational risk events add another $115 billion to the total loss figure.

Unlike the credit and market risk losses – which were generated specifically from an individual institution’s own data – the operational risk numbers reflect a total industry loss figure that was then parcelled out to institutions in proportion to their third-quarter 2011 Tier I common capital. They are therefore not broken down in the bank-by-bank results.

Evan Sekeris, assistant vice-president at the Richmond Fed, worked on this part of the project. He says the team focused on modelling event types that seemed to have a link to the macroeconomic environment for the purposes of the stress tests. So, for example, it didn’t model ‘damage to physical assets’ as “we weren’t expecting the model to show any kind of a macroeconomic link there,” he says.

The team used a panel regression technique to model the frequency of losses in the industry by event type because of the limited amount of operational risk data available for modelling. “Most banks and their data sets only had one severe downturn, which is the current one,” says Sekeris. “Few banks had data that covered the earlier downturn of the early 2000s, plus that downturn was fairly mild. So if one is looking to run a regression with a single bank, one is left with a single downturn, and it is really hard to come up with any meaningful statistical results there. The advantage of the panel is that if you have 19 banks, you have 19 downturns in your dataset. That is why we used the panel regression.”

The outcome of the stress-test work, according to Sekeris, could be the first step on a new, revolutionary road for operational risk. “There used to be this perception in the industry that operational risk is an acyclical risk,” he says. “I think this is a misconception in a way, driven by the fact people think of op risk losses as random acts of God, and so forth. And some of the big events do have a random component to them. But what we have identified here is that there is a cyclical component to operational risk. We’ve debunked that myth, if you will. I’m not going to say that this is the end of it. We are at the first stages of this kind of work, but the evidence we saw in the data this year clearly indicates there is some kind of cyclicality in operational risk.”

Sekeris says his team’s work found the most correlation with macroeconomic factors in the internal fraud, external fraud, and clients, products, and business practices operational risk loss event types. He adds he’s not in a position to speculate why these three loss types saw the most correlation, and that more work needs to be done.

The Fed team’s work is controversial. A significant contingent of operational risk executives and experts say the existing advanced measurement approach (AMA) – with its 99.9% confidence interval – is effectively a stress test, so the stress-testing programme is simply duplicative.
“If you measure operational risk at the 99.9% confidence level, with a one-year time horizon, you are measuring what is roughly equivalent to a one-in-1,000-year aggregate loss. Wouldn’t you define that to be a highly stressed situation?” asks Ali Samad-Khan, president of Stamford Risk Analytics. “To put this in perspective, consider the worst thing that’s happened to your organisation over the past 10 years. A one-in-1,000-year single event would be an extrapolation that was 100 times further out in the tail. Is there any conceivable economic need to know a situation worse than that?”

Samad-Khan adds: “One reason there is a perceived need to calculate stressed value-at-risk is that most models, particularly market and credit risk models, systematically underestimate risk. We know this because the supposed one-in-1,000-year event seems to take place every 15–20 years. This is because instead of calculating a true one-in-1,000-year event, most models actually calculate a one-in-20-year event, but represent this figure as a one-in-1,000-year loss. This happens when the assumptions underlying a model are flawed. And one of the most critical assumptions is homogeneity – for a model to be valid, the underlying data must be homogenous.”

Another problem is the lack of a robust data set. Stress testing operational risk is “a far more difficult exercise than for market or credit risk”, says Bahram Mirzai, a managing partner at risk management consultant EVMTech in Switzerland. “For most of [the market risks and credit risks] there is daily data. For macro variables such as GDP, there is the monthly data, but it is on a consistent basis. For operational risk losses, devising an index is not a straightforward exercise. You can’t just take the number of losses or the average loss per quarter or per annum and say ‘this is my index’. You need to undertake research to devise a meaningful index. Even then, in the best case there would be eight years of data and all that is being covered is one crisis.”

Many believe operational risk simply doesn’t seem to show strong correlations with macroeconomic factors. “It is the specific risk that brings a company down, not the systematic risk,” says Mirzai. Big operational risk losses are the product of individual events, such as a rogue trade or a class-action lawsuit, he points out. On the other hand, severe credit and market risk events have their causes in recessions, market crashes and other related economic events “This is a big difference between operational risk, and credit and market risks,” says Mirzai. “For credit and market risk, there are dependencies, or concentration risk, which result in big losses. For operational risk, it is individual events that result in big losses.”

Others say they have done extensive work themselves, and don’t see the correlations that the Federal Reserve team claims to have detected. Carsten Steinhoff, head of operational risk at Norddeutsche Landesbank, and Marcel Monien, quantitative risk expert at Norddeutsche, have researched macroeconomic correlations with operational risk events through the DakOR external loss consortiums, and Monien is also a member of the Institute of Operational Risk’s project group on scenario analysis and stress testing. At an IOR workshop in Frankfurt in March, attended by bank representatives from Germany and Austria, the group concluded the only remarkable correlation they found was a lagging correlation between an economic downturn and legal risk, in the form of lawsuits related to financial products.

Monien says that they did not see empirical correlations around either internal or external fraud and the economic downturn. Rather, they suggest, fraud types such as credit and debit card fraud, or internet banking fraud, are correlated to weak technology systems, or overall systems and controls. IOR has a dedicated subgroup that is working on a guidance paper regarding scenario analysis and stress testing. The paper is part of a continuing research programme – last year, the organisation published two papers, on risk categorisation and external loss events.

The interest in stress testing in Europe is also driven by regulators. In 2010, the Committee of European Banking Supervisors (Cebs) – the predecessor to the European Banking Authority – published its revised guidelines on stress testing, which included a section on operational risk. This paper was, in turn, generated by a global interest in stress testing that was driven by the Group of 20 advanced economies (G-20), which mandated stress testing as an important tool for regulators to use to try to understand systemic risk, as well as risk within individual institutions. Now, most firms are expected to perform stress testing within their Pillar II work on operational risk.

But most regulators have not published any additional guidance on stress testing in their own countries. Some people spoken to for this article say the lack of regulatory guidance reflects the fact the supervisors themselves don’t know what to suggest banks do for stress testing operational risk. Others say it’s simply a continuation of the ‘let a thousand flowers bloom’ strategy supervisors have often pursued. “It does appear that for banks, the position of the regulators around the issue of stress testing is similar to the early stages of the AMA,” says Jane Yao, vice-president at the American Bankers Association in Washington, DC. “Even the regulators are trying to find what the best practices are. They want them to evolve and emerge, rather than have a set idea about them already.”

There has been no guidance from the German regulators that takes the Cebs/EBA paper further, for example. “We try to understand a way to make this useful for us,” says Steinhoff. “For example, one type of stress testing for us is to stress the model parameters, so we see it as a sensitivity check for the models. The second is that we deal with the external data, and we try to check single data points with our model to see whether the model is robust. If there are lags in our control environment, if we have to take measures against these certain cases. But this is mainly driven by the external data.”

Like the US Fed, the UK’s Financial Services Authority (FSA) also expects firms to use “stress testing to estimate incremental expected loss over a forward-looking capital-planning horizon”, says Laurie Mayers, manager of the capital management team at the regulator. While she acknowledges it is early days for operational risk stress testing, she adds: “All risk types need to be further stressed to determine the incremental expected loss amount. The main point is that, when doing forward capital-adequacy stress tests, the whole firm has to be stressed, including all risks to which it is exposed – so op risk one way or another has to be included too.”

It is possible firms are simply looking for operational risk correlations in the wrong place. One recurrent criticism of stress testing is that the losses are not correlated to economic cycles. “How did the Bernie Madoff losses get exposed?” asks Samad-Khan. “They got exposed when people started liquidating their funds and it turned out that there wasn’t any money to be paid out. There was no cause and effect here. It wasn’t as though the fraud event was caused by the downturn in the economic cycle. The fraud had been taking place all along. Instead the economic downturn turned an unrealised loss into an observed loss. But this loss would have been discovered sooner or later because every Ponzi scheme eventually collapses under its own weight. So much of what people talk about when they talk about correlations in operational risk has to do with how they observe the losses, but not with what is really going on.”

This same argument also applies to rogue trading, for example – the positions build up during boom periods, when it is perhaps easier to hide losses, and are exposed during market reverses. But it is not the market reversal that caused the fraud in the first place.

Yakov Lantsman, a principal at Deloitte in New York, argues this is precisely the point. In a November 2007 white paper he co-authored with Penny Cagan while they were both at Algorithmics, The cyclicality of operational risk – the tracking phenomenon, the two authors argue that operational risk is correlated not with downturns but with the periods of volatility that precede them.

“Op risk losses are not related to decreases in economic activity, but are the result of increased economic activity prior to the recession,” says Lantsman. “There is a lag in how these operational risk losses show up.” He adds that, while credit risk and market risk losses are directly correlated with lower economic activity – people default because of lower incomes, portfolios shrink because markets decline – operational risk “is a different animal, which is why people don’t understand its anatomy”.

“Big operational risk losses – tail events – are not a reflection of a drop in business activity. They are a reflection of prior exuberance,” he continues. As a result, “stress testing does work if you take into account the historical period”.

Therefore, argues Lantsman, it’s possible that stress testing could be more helpful than the AMA for firms seeking to better understand the origination of the risks they face. Another place where the Basel II AMA framework falls down is around the fact that it only asks firms to model for one year, says Lantsman. Others agree performing the stress test over a period of nine quarters is valuable. “I think the stress-testing exercises are much more valuable, because they look at a two-year time horizon and ask not just whether a firm can withstand a severe shock, but also whether it can dig its way out of the hole and recover the capital fast enough,” says a senior operational risk executive at one of the 19 banks required to undertake the stress-testing programme in the US. “I think it is a flaw in the Basel II model, in that the regulators only look at one year’s sustainability, and then they magically assume the capital will just reappear, and that’s not how it happens in the real world. The stress testing is a better exercise because it forces you to look out over a multiple-year horizon.”

Lantsman also argues that understanding the systemic nature of operational risk by exploring stress testing is only half the answer. Firms need to better understand their own idiosyncratic risk as well, he argues. While there might be general correlations that can be understood, he says individual organisations will have specific factors that will determine how these correlations are exhibited. For example, an internal fraud might have been made worse by a particularly poor technology security system, or there might have been a change of leadership in a part of the organisation, leading to gaps forming in the product-approval process.

Lantsman’s point can be demonstrated by the stress-test results at Capital One, one of the 19 banks involved in the CCAR project. A senior operational risk executive at the bank says: “When the macroeconomy is getting worse – credit risk is increasing, gross domestic product is dropping, home prices are dropping, unemployment is rising – we see that execution risk improves. And we found internal fraud risk decreased as well. The only thing we could think of is that our business lines slow down, so when people stop buying and credit losses start mounting, the operations move into a ‘protect what we have’ mode. They get aggressive about controlling credit risk, they stop coming out with new innovative products, and it slows down the amount of change in the operating environment. All of that slows down the amount of errors we are seeing.” In short, Capital One’s risk response to an economic contraction resulted in an improvement in its operational risk position.

Sekeris acknowledges that results such as Lantsman’s throw doubt on the idea of a simple correlation between economic stress and operational risk, but says he is keen to better understand how the inner workings of individual firms affect the strength of the larger-scale correlations between operational risk event types and macroeconomic factors. “We want to go back and work more on the data and refine the models,” he says. In particular, he wants to further develop severity modelling, understand firms’ idiosyncratic risk, and dive deeper into what drives the correlations he has seen.

Sekeris adds that stress testing could help push forward a nascent discipline within operational risk – factor-driven models. “I hope we will be able to share a little bit more and help move the discussion forward, because in a way it ties into factor-driven models. This is a factor-driven model – we are trying to explain changes in the frequency of operational risk events using factors,” he says, adding it includes both macroeconomic factors and bank-specific factors, such as assets, number of employees and assets per employee.

Lantsman and others are keen for this work to develop further. Lantsman says: “We have a lot of data right now but it is not taken seriously. If op risk is an element of this stress testing, which goes to the board and to the final capital number, it will then have an impact.”