Broken backtests leave quant researchers at a loss

As historical data loses relevance, quants must find new ways to validate their theories

Investment prospectuses famously contain the standard disclaimer that past performance is no guarantee of future returns. It’s a rider that is usually ignored – even by those in the industry.

“The truth is, most investors treat past performance as the one and only guide to future returns,” says Anthony Morris, who heads quantitative strategies at Nomura.  

Quant strategies in particular are built using backtests, which show how they would have performed during long tracts of history.

Backtesting, quants concede, is at the heart of what they do. It’s the “gold standard”, says one, for gauging whether a new strategy or research idea is worth pursuing.    

The practice has always had its detractors. A global head of risk at one asset manager says he skips the final page of quant presentations, which usually end with a graph of backtested performance – always positive – as if it were undeniable proof of a strategy’s soundness.

Covid-19 and its effect on markets have added to those doubts. The implicit assumption behind backtesting – that the past includes periods that will resemble the future – may no longer hold. That realisation has left some quants feeling rudderless.

Risk.net asked one quant what they would do if historical data ceased to be informative. “Actually,” he says, before pausing, “that’s a great question”.     

Alexander Lipton, an academic and co-founder of fintech Sila, is more blunt. “Backtesting based on what was going on over the last 10 to 20 years is next to useless,” he says.

Some quants are already playing down their reliance on backtesting. At JP Morgan, credit structurer Danny White employs the same techniques used by the bank’s trading desk – including stress tests and Monte Carlo simulations – to understand how strategies might behave in unexpected conditions. 

The truth is, most investors treat past performance as the one and only guide to future returns

Anthony Morris, Nomura

“What happens if spreads widen by 100 basis points?” says White. “What happens if the curve steepens? If the curve inverts? What happens if spreads go 10% wider or the widest names in the portfolio default?” Backtests cannot provide all the answers.

At Barclays, the quant investing group built an algorithm to show how investment strategies will react to big, sudden changes in interest rates, currencies and inflation.  

Others are reviewing whether the theories behind backtested strategies are still valid. Quants at Aspect Capital spent the summer re-checking assumptions, strategy-by-strategy. “We can never be certain we’re correct. But what we can do is check there’s no new information that means a hypothesis cannot be correct,” says Christopher Reeve, director of risk at Aspect Capital. “We want to avoid trading where something has changed in the real world – be it a yield curve effect or a central bank effect – so that a relationship we think was true can no longer be true.”

Lipton thinks more fundamental changes are required. He urges quants to embrace a more iterative approach, similar to that of physicists, where theories are developed and fine-tuned in a series of controlled experiments, rather than purely relying on historical observations.

“For quants, that means building theories and testing them in live trading, not with $5 billion, of course, but maybe with $50 million.”

A brief history of time

The history of interest rates provides the most vivid example of the problems with backtesting. JP Morgan’s government bond database, one of the most complete data sets available and a favourite of fixed income investors, starts in 1985. Interest rates in the US have declined steadily since peaking at 20% in October 1981. Backtests conducted with this data reflect rates taking this “entirely one-way trip”, Morris says – one that, by construction, cannot be repeated anytime soon.

This could have profound implications for investors. Sandrine Ungari, head of cross-asset research at Societe Generale, says it is conceivable that the monetary policy response to Covid-19 could break the link between stocks and bonds that has held since the late 1990s.

Katy Kaminski, chief research strategist at AlphaSimplex, says bond buying by central banks has already had “perverse effects” on markets – the recovery in stocks despite plunging economic indicators being the clearest example.  “It definitely feels like markets are going to move in places we don’t expect them to,” she says. “I worry that something is going to give.”

Katy Kaminski, AlphaSimplex

So, what’s a quant to do when the data isn’t relevant? One option is to reach further back in history to try and capture a wider range of regimes. Farouk Jivraj, head of quantitative investment strategies research at Barclays, says the group tries to backtest as far back as the data allows – to the 1800s, in some cases.

In another example, AQR backtested a trend-following strategy over 110 years to prove its consistency, while Winton has charted the price of cocoa over 200 years.   

Others say extending backtests might be of little help at this time. “In some cases, 50 years of data is worse than nothing,” says Nomura’s Morris. “If that sample period is telling you something that’s different from the current reality, it’s worse than useless.”

Another workaround is to use samples that most resemble the current environment – for instance, data from Japan to test the effect of near-zero rates on US assets. But this approach also carries risks.    

“It does help but also it doesn’t,” says Milind Sharma, founder of hedge fund QuantZ Capital Management and a former prop trader at Deutsche Bank and RBC Capital Markets. “It might give you false comfort, given the implicit assumption that conditions now are comparable. It’s too short a sample to control for all the other variables.”

In other words, quants can’t be sure how much of what they see in the backtests is due to low rates versus other conditions specific to Japan at the time.

It definitely feels like markets are going to move in places we don’t expect them to

Katy Kaminski, AlphaSimplex

Some have all but given up on backtesting their more imaginative ideas. Melissa Brown, head of applied research at Qontigo, recommends sticking with simpler strategies based on sound rationales that have performed well through different regimes in the past.  

“Obviously, nothing’s going to work all the time,” she says. “But if there is a strategy that seems justified in different kinds of environments – given the uncertainty going forward, that’s a better strategy.”

A strategy such as trend-following, the thinking goes, captures a truism of investor behaviour so fundamental that it will reliably persist. That is, investors will favour rising assets over ones that are falling, no matter what’s happening in markets more widely.  

QuantZ’s Sharma takes this reasoning a step further. Risk aversion and risk-seeking behaviour manifest in “surprisingly predictable” ways in equity factors, he says. His firm’s risk premia strategies are designed to do well in intense risk-on and risk-off environments, when fear and greed are most pronounced. Counterintuitively, the behaviour of investors during tail events is easier to predict and monetise, he argues.

“The next crash will likely be triggered by a different catalyst. But that doesn’t matter. In a crisis, sentiment drives the market,” he says. “Assets seen as flight-to-safety havens are going to rally, regardless, while vol stays elevated. It doesn’t matter if rates are down to zero. Treasury bonds will still see a bid.”

Trial and error

Some think quants will have to go further and fundamentally change how they come up with ideas. They describe a shift to a more opportunistic, more adaptable approach – being ready to implement trading strategies that may not stand out in past data or perform so well in backtests.

As Chloi Karyda, a structurer in the investable index solutions team at JP Morgan, points out, strategies that control for explicit duration exposure – which may fare better in the future – will almost by definition look weaker in backtests from a yield diversification perspective.

Nomura’s Morris says quants will need to be more creative and use their market knowledge to find new opportunities. “Just because you have quant tools doesn’t mean you can turn off other parts of your brain,” he says. “We’re trying to look in areas not covered by thousands of other people and that are less likely to have been tapped. You need something original, a different angle, a different edge, something that plausibly hasn’t been understood or captured before. That’s not easy to find.”   

The focus needs to be on “what works” today, Morris says.

Alexander Lipton

Lipton thinks the future of quant investing will rely more on trial and error. “We need to bring the discipline much closer to what physics did at the apex of its progress,” he says – an iterative process in which practitioners refine ideas in stages through experimentation.

Quants will need to scout out different data, Lipton says, and use it differently too. They must employ pockets of historical information rather than entire histories to check a hypothesis. He refers to this as “historical testing”.

Lipton once used data from Europe in the 1940s to understand how bonds trade during periods of military occupation. If researchers uncover something fundamental in markets, he says, it ought to show up in even in the most extreme periods or far flung geographies.

Lipton even advocates having a “historian in residence”, who can alert quants to past episodes that might confirm or confound their assumptions. “And I don’t mean how things were five years ago in London. I mean in Budapest in the 1930s or Tokyo in the 1940s.” Finding such arcane data takes time and money, he acknowledges. “You need to go down a lot of rabbit holes to get it.”

Machine learning and newer alternative data may also have a role to play. Lipton and Marcos Lopez de Prado wrote for Risk.net in April about how these could help quants find previously missed patterns and detect shifts in the macroeconomic environment in near real time.

Will quants change the way they work? That probably depends on what asset owners do. Some say clients are already favouring a more opportunistic approach.

Toby Goodworth at bfinance advises institutions on quant fund selection. Flows into quant strategies have been growing in recent months, as investors look for alternatives to traditional fixed income, he tells Risk.net.

At the same time, clients are seeking more adaptability. “They’re choosing multi-strategy relative value funds – not a specific credit arbitrage, not just equity market neutral, not just merger arbitrage, but all those things combined in a more tactical approach.”

In the face of uncertainty, Goodworth says, clients want an approach that allows for “more responsive tuning”. Cleaving to the past, it seems, is going out of fashion.

• Tweet  
• Facebook  
• LinkedIn  
• Save this article
• Send to  
• Print this page  

• Topics
• Backtesting
• Quantitative analysis
• Quantitative finance
• Research
• Quant investing
• Covid