Covid scenarios: finding the worst worst-case

As pandemic trashes historical data, a Risk.net tie-up with Ron Dembo’s new outfit tests promise of polling

This is the first in Risk.net’s series of crowdsourced scenario-generation exercises.

By September 26, the still-spreading coronavirus will have extracted a heavy toll from the US economy, and triggered a catastrophic collapse in demand for US assets. The dollar will have lost 20.25% against the euro and 25.5% against the yen; the S&P 500 will plunge by a third. Meanwhile, the 10-year US Treasury will be yielding 0.1%.

It’s a bleak, bruising scenario – it may not seem very likely today – but that’s precisely the point.

The scenario is one of 64 derived from a survey of more than 300 Risk.net readers, carried out in late March. Their six-month forecasts of individual risk factors were knitted into multi-factor scenarios by RiskThinking.ai – the start-up identified and combined the population’s more extreme views.

To put it another way, this is crowdsourced stress – and it’s an attempt to tackle one of the key challenges that arises during periods of pervasive uncertainty, when backward-looking risk models are rudderless and the traditional fallback is to ask small groups of in-house experts for their judgement on how bad things could plausibly get.

That fallback is where many portfolio and risk managers currently find themselves, and some are not enjoying it.

A senior quant involved in scenario design at one US bank describes scenario construction for the pandemic as “an excruciatingly impossible task – I mean, I really, really don’t like it”. His beef with the process is that the bank is attaching finger-in-the-air loss estimates to a range of scenarios, rather than discussing and analysing its vulnerability to those scenarios.

A simpler objection comes from a former regulator: “I think most people within banks haven’t got a bloody clue how bad this is going to get.”

At one large Asia-Pacific bank, the regional head of model validation describes having to make Covid-enforced adjustments to a sea of indicators – from probability of default modelling for loans, to net interest income models – because the bank expects the eventual impact of the virus to be far worse than the worst-case losses implied by any recession in its dataset.

I think most people within banks haven’t got a bloody clue how bad this is going to get

Former regulator

“Right now, we’re in a unique situation: a lot of the analytical tools and models that are seen as business-as-usual in stress testing are either not relevant, or giving outputs that don’t make a lot of sense, simply because of the unique movements we’ve seen,” he says. “So, you’re going to have a lot of uncertainty with the traditional models you’ve got in place – that’s understood. Management decision-making and overrides, a revisiting of assumptions, is a key process that’s currently underway.”

Survey results and further reading

Click here for the results of the pilot survey

A downloadable spreadsheet containing the scenarios is available here.

An interview with Ron Dembo is available here.

A paper describing the scenario-construction approach can be found here.

In that context, he suggests, simply asking a large number of people how bad they expect things to get seems a reasonable exercise.

That’s where the survey comes in. Polling outsiders is not the way stress scenarios are normally constructed, and some dismiss it as a solution. Ron Dembo – founder of RiskThinking.ai and, in 1989, of modelling vendor Algorithmics – argues it’s an idea whose time has come. Even before the pandemic injected uncertainty into a host of critical risk factors, market participants were struggling to work out their exposure to climate change, to cyber crime and to technological change. Those struggles will continue once the questions associated with Covid-19 have been answered.

Outlier views

The predictive capacity of the pilot survey can already be tested.

Respondents in Asia-Pacific expected, correctly, to be locked down for less time than their peers in the US and Europe. Risk managers were marginally less pessimistic about the ensuing first-quarter hit to US GDP than their peers in banking, broking and consulting, but everyone expected it to be pretty bad: a median average of all those surveyed predicted a 5% drop in GDP – significantly worse than the 3.5% being predicted by economists at the time, and almost bang on the actual 4.8% fall published by US statisticians last month. And come the end of September, almost all respondents expect the S&P 500 to be some way below its March 26 closing level of 2,630, with a 15% drawdown the average view. The index is currently hovering around 3,000.

The point of the exercise was not to find a consensus, however: the shape of the distribution of responses is the important part – and, within them, the extremes weighted against the overall distribution. These outlier views serve as the inputs for stress scenarios that are built from the survey, based on combinations of multiple factors arrived at via a decision tree analysis.

Dembo acknowledges the approach is not without controversy – particularly, given the importance placed on them, the crucial process of distinguishing between sincerely held outlier views and mischief-making. The former are plugged into the scenarios, while the latter is discarded. As Dembo puts it: “It’s an art, weighing up whether someone is trolling us, or for real.”

Even as a method of reliably spotting black swans, “people will find it controversial, and we accept that”, Dembo adds.

He’s not wrong. Those used to being given a scenario expressed in terms of its impact on financial markers express bewilderment at being asked to provide the opposite.

“How do you ‘detect’ something that’s used as input?” asks the chief risk officer at a large European asset manager. “A black swan is an unexpected event. If you start by defining an event, and then ask experts how it impacts markets, your model doesn’t ‘find’ black swans. It is the input of the model, not the output.”

Dembo responds: “If I want to find genuine black swans, I need an extreme range of views. We as individuals are pretty bad at generating extremes. But on single factors with a diversity of opinions, we might uncover more extreme views than the consensus. We are completely useless at dreaming up scenarios on multiple factors, however. That’s why we poll a broad range of experts and seek well-justified extreme views.”

“We believe people are much better at capturing that uncertainty in a single factor,” he adds.

He contrasts the approach with the way scenario analysis is usually conducted in banking, wherein an end-state is pre-defined – either by dedicated teams within the organisation, or set by its supervisors – and the lender then sets about decomposing its impact on its portfolios and loan books, to work out its all-in exposure, usually as a means of assessing capital adequacy to those events.

Stay involved

In the coming weeks and months, Risk.net will work with RiskThinking.ai on a series of follow-up surveys, publishing the results of the polls – and the scenarios – for readers. We expect it to provoke debate; we hope some of the results are useful.

If you want to join the crowd and contribute your own forecasts, sign-up here. Participants will receive the results prior to their wider publication.

Let us know what you think. Please send any feedback to [e-mail]

That process itself, though based on quantitative techniques, relies a lot on expert judgement and layering of assumptions to arrive at an agreed impact estimate. More importantly, Dembo contends, if firms are using it to establish a true picture of their exposure to genuinely extreme events, then they’re going about it the wrong way.

There is some sympathy for that view. The Apac bank’s regional head of model sees polling as a way to benchmark or challenge the in-house views, rather than replacing them. “The scenarios give you a range of current market views, which – on top of what you’re doing anyway – will hopefully give you added visibility, and hopefully allow better financial decisions to be made, anticipating how many reserves to set aside for the credit books, things of that variety. From that point of view, it’s a good exercise to go through,” he says.

The distributions

A look at the distributions shows some respondents foresee further heavy losses across equity, bond and currencies portfolios.

 

 

Figure 1 shows respondents’ predictions for percentage gains or losses for the S&P 500, relative to its closing level on March 26 (2,630). A mean average of respondents predicted a 14% decline for the blue chip index – but a small group at the lower bound predicted a drop of more than 80%. (A few optimistic souls, currently closer to being in the money, predicted a 72% gain.)

A wide spread of responses might appear surprising, but Evan Sekeris, who previously oversaw op risk at the US Federal Reserve, sees this as a key advantage of the crowdsourcing approach: there is information in the shape of the distribution, not just in the extremes that are used to generate the scenario.

A relatively narrow distribution of responses – say, between -10% and -20% – would suggest less uncertainty on the topic being polled, he argues. “But if you have a distribution like you have here, where some people are saying it could go to minus 80%, while others are telling you it could double, or go up 50% or 100%, then that’s a completely different picture, right? Which is: nobody really knows. The consensus seems to be around this number – but there’s small sub-buckets – one in the positive and one in the negative – that strongly think otherwise. That information alone is very important.”

 

 

The picture is similar for expectations of the euro’s value against the dollar. An average of 303 respondents predict a 5% decline for the single currency against the greenback over the next six months – but those at the lower bound predicted an 86% decline, while those at the other end of the distribution expected a 63% gain.

Interestingly, while regional expectations were broadly in line with this picture, the distribution of responses from North America, consisting of 60 votes, was tighter: a mean average expected a 4.4% decline for the euro, while the most extreme predictions were of a 57% decline – and, at the other end of the scale, a 61% decline for the dollar. In Europe, the more extreme predictions of a 161 sample count were more severe: a mean decline of 3.1% was anticipated, with a median of -5%; but some predicted an 83% decline for the euro, while others saw a 58% gain versus the dollar.

This is surprising, says one leading academic on the construction of stress scenarios; ordinarily, with a larger sample size, a distribution pattern would tend to be tighter. In fact, with respect to the shape of the distribution’s tails, the opposite appears to be the case – again, lending support to Dembo’s premise that if one garners enough views, a statistically significant number of respondents in a sample will predict an extreme outcome.

“Usually by n=300, you should have a pretty normal distribution – and you didn’t have that, across a lot of things. You had a pretty decent sample size, and yet you had some pretty funky distributions. I was surprised you didn’t have a greater convergence around ‘normality’,” he says.

The shape of those distributions suggests “there’s a lot there to tease out about what is at the edge of people’s beliefs,” he adds. “Even if you just say ‘there’s always a few outliers’, that doesn’t explain why you still have, at one or two standard deviations, quite a number of professional investors and the like who believe something that seems to be quite outlandish, relative to current asset prices. That’s a train of thought that I think would be really useful for investors, for your readers, for policy-makers.”

 

 

The survey also highlights the starkly different realities respondents in different parts of the globe expect to be living with while the pandemic lasts. Asked how many weeks it would be until the majority of schools, businesses and other places of work are “functioning normally” in their own region, expectations were again wildly different. While the mean expectation was of an 18-week lockdown, a significant chunk of respondents said they expected it to last for 52 weeks – although here, much might have turned on respondents’ interpretation of “functioning normally”.

Viewed regionally, a clear split on lockdown expectations emerges between respondents in the US and Europe versus those in Asia-Pacific and the rest of the world; all four regional distributions are similarly shaped, with the mean expectation falling in a tight range between 17 and 20 weeks; but the positioning of the lower bound for each varies hugely, on a proportional basis: in Europe and the US, the shortest lockdown anyone anticipated in late March was four weeks; in Asia-Pacific, where South Korea avoided a complete shutdown of the economy altogether, the lower bound fell at one week.

The scenarios

(Scroll to the bottom to view the scenarios table.)

In Dembo’s conceit, it is the positioning of the lower and upper bounds that is crucial as a means of challenging consensus expectations; the stress scenarios designed to capture black swans – unlikely but plausible events combining in unexpected ways with hard-to-predict consequences – are themselves based on combinations of these extreme views, constructed via a decision tree analysis.

Sekeris agrees with that approach. He too argues banks should start by determining the path that leads to a loss or an event, by decomposing it – so, by finding the different variables in the decision tree that lead to that final outcome. The tree itself then becomes hundreds of final outcomes that are the various possible combinations of all the different nodes, and the different paths events might take.

“But the idea is that then when you go to the experts, you don’t ask them to tell you what the final outcome is. Because the point is, no matter how much of an expert you are, when you’re asking just the final outcome, you're basically asking them to mentally do all the calculations of that tree – to go through all these three steps and say, ‘Okay, if the S&P goes to here, the next step is, this happens, then this happens – and here’s my outcome.’ And what happens then is, you lose that all-important range of numbers at the end, because they’ll say, ‘our worst-case expectation is a loss of X’. Well, plus or minus how much? It’s never going to be a loss of exactly X,” he says.

Worse, he adds, the way the expert arrived at that loss figure is highly likely to be prone to error and bias – a product of the computational shortcuts that someone is taking in trying to come up with that final number, without focusing on the elements before.

“What you’re doing here with your structure is forcing people to ignore that final number and to focus on the elements of the tree, the nodes. And that’s very important, because those nodes are things that they understand better, and they can provide you information much more clearly about,” he says.

Of the scenarios contained in the decision tree, 64 are then combined into a table, the raw data for which can be downloaded here. Each is assigned a likelihood score, ranging from 2.99% to 0.73%.

The scenario with the highest likelihood – number 47, on the table – posits the following: a 12-week lockdown; a 7.5% gain for the dollar versus the euro through September 26; a 2.5% gain versus the yen; a 25% gain for the S&P 500; a yield of 0.9% on US 10-year bonds – and a -21% first-quarter hit to US growth.

The least likely scenario – summarised at the top of this article – encompasses: a 30-week lockdown; a 20.25% decline for the dollar against the euro; 25.5% decline against the yen; a 33% decline for the S&P 500; a yield of 0.1% for US Treasuries; and a 3% first-quarter decline in GDP.

At press time, scenario 47 certainly appears more true to life: most G20 economies remain in some form of lockdown, though roughly 12 weeks on, many of the conditions attached are steadily being relaxed; the euro is currently up 1.3% on the dollar, while the greenback is up 21.3% on the yen. The S&P is currently up 15.7%, while the yield on US Treasuries stands at 0.65%. US GDP declined by 4.8% during the first quarter; current estimates put the hit to Q2 GDP anywhere between -12% and -40%. In other words, as Sekeris says, “nobody really knows”.