Q&A: Ron Dembo on crowd-spotting black swans

Veteran quant argues large groups are better at gauging extreme uncertainty than small teams of experts

Data

This article accompanies Risk.net’s project on crowdsourced scenario generation.

The future isn’t what it used to be – at least, not if judged by the financial industry’s efforts at modelling it.

Time and again, the standard method of estimating losses by looking at what has happened before – imagining the future by selectively replaying the past – has been found wanting.

Covid-19 is just the latest example. For every day that passes with the world’s largest economies under lockdown, the outlook darkens – tens of billions of dollars in loan losses, double-digit hits to growth, and runaway unemployment.  

Could banks have seen any of this coming? Ron Dembo thinks so.

The veteran quant – a former Yale professor turned serial risk software entrepreneur – has a new venture, RiskThinking.AI, which is trying to bring about a mindset shift in the field of scenario generation.

The basic aim of scenario generation is to build a forward-looking gauge of risks for which there are few or no precedents to rely on. Various quantitative approaches are used to estimate losses by working out what could go wrong, and how bad the consequences could be. The outputs of the analysis are then used to put a dollar value on a firm’s risk exposures.

But for events whose impact carries a high degree of uncertainty, such as climate change, cyber risk or indeed pandemics, Dembo argues that the classical approach – using a small cadre of in-house experts to work out what could go wrong, and then working backwards to see how the firm could be affected – is misguided at best, and dangerously misleading at worst.

By seeking enough expectations, you can derive extreme events. When you have extreme uncertainty, you want to capture that – that’s the ultimate driver of our scenarios

Ron Dembo, RiskThinking.AI

His instinct is instead to trust in the wisdom of crowds: polling a broad audience of risk and finance professionals – Risk.net readers, in the case of a recent survey on Covid-19 – to generate estimates of an event’s impact on a range of key financial indicators. These are then layered together in different combinations to form scenarios. The basic premise being, if one garners enough views, a statistically significant number of respondents in a sample will predict an extreme outcome.

Dembo accepts many in finance, particularly quants, will find the approach hard to stomach. But he insists it is the best hope firms have of getting a handle on risks that classical statistical techniques probably can’t handle.

“The point we’re making is, by seeking enough expectations, you can derive extreme events. When you have extreme uncertainty, you want to capture that – that’s the ultimate driver of our scenarios. We believe people are much better at capturing that uncertainty in a single factor,” he says. “This is not the way people normally construct scenarios. People will find it controversial, and we accept that.”

Ron Dembo
Ron Dembo, RiskThinking.AI

As evidence, he points to how few firms or governments anticipated the impact of Covid-19. Yet pandemics have happened before, many of them more deadly, he points out. If a firm had cast the net wide enough in January, it could have captured the outlier views of those who believed the pandemic could spread rapidly, that countries would enter lengthy lockdowns, and that extreme economic damage would be the result.

“If you’d just polled epidemiologists, they’d have said: ‘Here’s one possible scenario: this thing goes nuts, and flies all over the world.’ You can extrapolate from that: ‘What happens if we have to shut down entire economies?’ But the question is: would you have extrapolated from that starting point and given enough thought to the potential consequences without polling a broader number of financial professionals?”

A tricky part of Dembo’s approach is weighting the extreme possibilities suggested within a broad sample, giving an indication of how likely respondents believe they are to occur. This is achieved by weighting outlier views against the overall distribution of the results.

Before that, though, the firm’s quants must apply judgement to distinguish between genuine outlier views, and people not taking the exercise seriously – an approach they acknowledge is open to accusations of bias.

“It’s an art, weighing up whether someone is trolling us, or for real. Sometimes, when you look at the responses, you see 100% changes for a given factor. Sometimes, we take those into account – because they’re not crazy. But when you look at distributions and see a 50% change in a currency – when did we last see that in the euro in a few months? It couldn’t happen. But it is an indication of the wild extremes people expect. You can then weight the responses any way you like, but the reality is there’s a real person who said this – and in the case of the poll we conducted with Risk.net readers, they probably do work for a bank.”

Why is polling large groups of professionals capturing better response to extreme uncertainty than asking a small group of in-house experts?

Ron Dembo: Starting with first principles: why do banks and financial firms generate scenarios? They do it to somehow get their arms around future uncertainty. No-one believes most of the scenarios they build will come to pass – they are there to guide future strategy in the face of uncertainty.

But there’s a fundamental misunderstanding of scenario generation within banks and public bodies that it’s just another form of forecasting, one that can be done by a small cadre of dedicated experts. It’s not. In the biggest of banks, in the highest of positions, scenario thinking is not well understood. To quote one such discussion I had, the person in charge said: “We generate many scenarios, and then select the best one.”

Why is asking more people necessarily a better guide? Don’t you just arrive at a broader market consensus?

RD: If I want to find genuine black swans, I need an extreme range of views. We as individuals are pretty bad at generating extremes. But on single factors with a diversity of opinions, we might uncover more extreme views than the consensus. We are completely useless at dreaming up scenarios on multiple factors, however. That’s why we poll a broad range of experts and seek well-justified extreme views. And that’s why we have developed an algorithm for combining individual factor uncertainty into scenarios on multiple factors.

For example, the recent bushfires in Australia were caused by a combination of unusual heat, high winds and drought. That’s a particular combination – if you’d just looked at the likelihood of each of those in isolation, then you wouldn’t have uncovered useful information. Estimation of multi-factor black swans is the challenge.

But the black swans you’re looking for are based on a combination of the single-factor inputs offered by survey respondents. How does your approach detect a black swan from single factors?

RD: The inputs are the uncertainty reflected in those single factors – scenarios constructed on combinations of multiple factors are the outputs.

Put another way, when is a black swan a black swan? What if I’d told you, in December, that some observers had detected an outbreak of some sort of strange flu in China? As it ripples out across the rest of the world, there’s a small chance that’s the start of a pandemic. It’s not like you’d never have thought a pandemic could spread in such a way. But if you’d asked anyone then to predict what we’re experiencing now, they’d have said: “Oh, that’s a black swan.”

Yet if, in January, you knew there was a virus in Wuhan, and you’d then looked at data from the 1918 Spanish flu, you’d have been able to construct a scenario that looked a lot like what we’re facing today. If you’d just polled epidemiologists, they’d have said: “Here’s one possible scenario: this thing goes nuts, and flies all over the world.” You can extrapolate from that: “What happens if we have to shut down entire economies?” But the question is, would you have extrapolated from that starting point and given enough thought to the potential consequences without polling a broader number of financial professionals?

The point we’re making is, by seeking enough expectations, you can derive extreme events. When you have extreme uncertainty, you want to capture that – that’s the ultimate driver of our scenarios. We believe people are much better at capturing that uncertainty in a single factor. This is not the way people normally construct scenarios. People will find it controversial, and we accept that.

Even if you had polled a broader number of financial professionals and concluded this was one possible outcome, what probability would you have assigned to it?

RD: It depends how the views that informed that scenario compared to the overall shape of responses – and how you then weight the distribution of views above and below the consensus value. That’s what’s reflected in the ‘likelihood’ column on the scenario table (we prefer to use the term ‘likelihood’ because, in radically uncertain situations, people debate the use of probability).

However, the important thing is to ask: “Do I want to ignore this scenario or take it into account, independent of its likelihood?” If I ignore it, then I am taking an out-and-out bet that something similar will not happen. If I decide to take it into account, then I will need to ask the question: “What is the hedge?”

How are the upper and lower bounds set?

RD: We’re looking for the worst-case scenario, so we take the extreme values in that range as the upper and lower bounds. Zero is the consensus value – either side of that, you find the range of impacts. When we look at both tail distributions – and if we’re looking for the worst- and best-case outcomes – my contention is we should choose the extremes: we choose the lower bound from the lower part of the range, and vice versa for the upper part. However, these extremes must be feasible at the very least.

We’ve now got a distribution that captures future uncertainty. All that interests us at this point is an upward or downward move – Brexit happening, or not, for instance. We’re only interested in a binary outcome. Now, what value do we give to the upward move at a node in the binomial tree? I need four numbers at each node: I need the value up or down, and I need the probability of each.

The upper and lower bounds are ultimately found by making sure we do not include any values that are not possible.

Usually, when constructing stress scenarios, a lot of emphasis would be placed on the correlations between factors. Why does your approach not consider correlations in this way?

RD: When people stress-test, they usually make certain correlation assumptions. In the real world, when stresses occur, correlations go out the window. Unless you’ve got a way of stressing covariance matrices, which I’d like to see, I don’t know how you can use correlations in a stress test. Correlations are also inherently backward-looking. What we’re trying to build is a forward-looking measure – going back and working out correlation is pretty much the antithesis of what we’re trying to do here.

What the approach offers instead is implied correlations. When you look at the scenario tree, it implies a correlation matrix. We might want to assume that, for instance, if the US 10-year rate moves, it will affect the S&P 500. So, you can use conditional probabilities on the branches – in other words, what’s the conditional probability of one usually moving down when the other moves up?

What number of respondents is considered statistically significant in order to get the desired distributions?

RD: Much work has been done on this in the field of expert elicitation. In many cases, when constructing scenarios now, people use samples as low as 40 respondents. The challenge is to get good estimates of the extremes.

Doesn’t a human input constitute a human bias by definition, contradicting the intent of minimising such biases?

RD: Human input is definitely human bias – but that bias is minimised by looking at a very broad range of opinions on single factors. We can’t eliminate human bias, but we can minimise it by combining factors in a given scenario using our algorithm.

How do you differentiate between the kind of extreme views you need to capture in order for this to work and someone trying to game the outcome?

RD: It’s an art, weighing up whether someone is trolling us, or for real. Sometimes, when you look at the responses, you see 100% changes for a given factor. Sometimes, we take those into account – because they’re not crazy. But when you look at distributions and see a 50% change in a currency – when did we last see that in the euro in a few months? It couldn’t happen. But it is an indication of the wild extremes people expect. You can then weight the responses any way you like, but the reality is there’s a real person who said this – and in the case of the poll we conducted with Risk.net readers, they probably do work for a bank.

We use expert judgement to remove genuine outliers. As an example of this, some respondents predicted a value of zero for the S&P 500. Clearly, this is nonsensical, so we eliminated it. Things are not always so black and white, but a close look at the extremes can eliminate impossible outcomes, or outcomes that lack any theoretical basis. Clearly, this can introduce some bias if it is not done carefully. But, certainly, elimination of events that have never happened before in recorded history will not necessarily be a good criterion for elimination.

Only users who have a paid subscription or are part of a corporate subscription are able to print or copy content.

To access these options, along with all other subscription benefits, please contact info@risk.net or view our subscription options here: http://subscriptions.risk.net/subscribe

You are currently unable to copy this content. Please contact info@risk.net to find out more.

You need to sign in to use this feature. If you don’t have a Risk.net account, please register for a trial.

Sign in
You are currently on corporate access.

To use this feature you will need an individual account. If you have one already please sign in.

Sign in.

Alternatively you can request an individual account here