# Quants tackle hedge fund operational risk

## Quantifying operational risk

Over the past three decades, the investment world has been transformed by an army of mathematicians, physicists and computer scientists using sophisticated quantitative techniques to play the markets.

Nowhere is this more apparent than in the hedge fund industry, which has embraced quantitatively driven strategies such as statistical arbitrage and managed futures.

Investors have also had to adapt. The best funds of hedge funds (FoHFs) have developed an array of sophisticated quantitative scoring systems and analytical tools to aid them in their investment decisions.

Yet one corner of the hedge fund industry has firmly resisted the advance of the quants. Operational due diligence (ODD) – identifying potential failures in a manager’s systems, people or processes – is still considered more of an art than a science, and decisions over whether to pass or fail a manager tend to be largely subjective.

“There is a general resistance to the whole idea of quantifying operational risk,” says Stephen Brown, a professor of finance at New York University’s Stern School of Business, who has written several papers on hedge fund operational due diligence.

There are a couple of good reasons for this. First, operational risk is inherently multi-dimensional. Systems, people and processes can fail in a variety of ways. The non-linear relationship between operational variables and fund failure is particularly difficult to model using traditional statistical techniques.

Second, on a more practical level, the data necessary to perform elaborate quantitative studies of operational risk is not readily available to academics or practitioners.

As a result “efforts to find a single variable to quantitatively capture operational risk have been met with a degree of scepticism in the industry”, says Brown.

However, as investors become more institutional and sophisticated, the idea of creating a more scientific and objective framework around ODD is gaining traction.

“The firms that are good at due diligence have typically taken a quantitative approach, with a qualitative spin. I don’t think you want to rely on a process that’s totally subjective,” says Josh Barlow, associate director of operational due diligence at Paamco, an $8.5 billion FoHF manager. Scoring systems for measuring operational risk have been around for some time. These are favoured by many ODD specialists because they promote consistency – ensuring that all hedge funds are measured according to the same criteria – and make it possible to perform peer analysis, track changes in a manager’s operations and weight different factors based on their relevance. Paamco separates a manager’s operations into 10 main areas such as pricing/valuation, counterparty risk, corporate governance and compliance – and uses a three-tier scoring system to assess the quality of each of these functions. Some of these functions can be empirically tested with a quantitative approach. Barlow points to pricing and valuation as an example. As Paamco receives position-level transparency from managers, it can systematically check for discrepancies in the way securities are priced across multiple portfolios. If a manager’s marks exceed a pre-defined variance threshold relative to peers, a red flag is raised, alerting the ODD team to take a closer look at the pricing procedure. In this example, Paamco’s approach to ODD “is pretty heavy on the science”, says Barlow. Pricing and valuation is one area of operational risk that is naturally suited to quantitative analysis. In the mid-2000s, Adil Abdulali, director of risk measurement at Protégé Partners, a$2.1 billion FoHFs manager, invented a quantitative measure known as the bias ratio, which helps spot instances of subjective pricing and return smoothing in hedge funds.

As a mortgage-backed securities trader in the 1990s, Abdulali had first-hand experience of how subjective the process of pricing illiquid assets can be.

“The idea that you can have firm and objective prices for assets that rarely trade is a fiction,” he says. “In reality there is a range of reasonable prices and you’re picking within that range.”

When Abdulali moved to the FoHF industry, he recognised that managers may be tempted to deviate from their pricing policies and mark positions at the higher end of the range if it turned an otherwise slightly negative month into a positive one – a phenomenon known as return smoothing.

“I started looking at the return distribution of various hedge funds and found this strange behaviour around zero, which was indicative of subjective pricing,” says Abdulali. “From there I was able to find a simple geometric formula to identify and measure this behaviour.”

The bias ratio measures the shape of the distribution of a manager’s returns around zero and compares this with their peers. Those with a comparatively high bias ratio are flagged, which alerts Protégé’s ODD team to check for potential return smoothing and the existence of illiquid or hard-to-value securities in the portfolio.

The critical threshold for the bias ratio varies depending on the manager’s strategy. The bias ratio of a perfectly valued portfolio of liquid, publicly traded securities should be close to one. An equity fund with a bias ratio of above two or three is considered suspect, while the threshold for less liquid fixed income and credit funds is around four. For structured credit funds a score of more than five is a definite concern.

Dan Federmann, chief operating officer and co-head of operational due diligence at Protégé Partners, says the bias ratio has proved to be a useful tool over the years.

“It’s great for identifying situations where we need to focus specifically on the subjectivity of the pricing process,” he says. “There are instances where the pricing policy seems perfectly fine on the surface but the bias ratio will indicate some form of geometric anomaly, which may prompt the ODD team to do more forensic testing around how the policy is applied in practice.”

The bias ratio is a tool rather than a hard and fast rule, adds Federmann. Protégé does not automatically fail managers that have a high ratio.

“There will always be some subjectivity around the pricing of certain securities. A manager may have a high bias ratio because they are particularly conservative around pricing and we may view that as positive,” he says. “That important point is that we’re aware of any subjectivity in the way managers price positions and how that impacts valuations in practice.”

The bias ratio is one of a handful of quantitative tools that have intrigued ODD practitioners. Others include formulas based around Benford’s law, which is used to spot falsified accounts and financial statements, and variations of the Altman Z-score, which measures the financial strength of companies. However, none of the FoHFs contacted Hedge Funds Review have formally integrated these models into their due diligence process in the way Protégé has adopted the bias ratio.

While FoHFs are exploring various quantitative techniques to detect potential problems in discrete areas such as pricing and accounting, NYU’s Brown has taken things even further.

In 2009, he published a paper with fellow academics William Goetzmann, Bing Liang and Christopher Schwarz, which argued that hedge fund operational risk can be expressed as a single number, called the omega score, in much the same way value-at-risk (VaR) measures market risks.

The paper directly addresses the multi-faceted nature of operational risk. In it the authors suggest a statistical technique known as canonical correlation analysis can be used to quantitatively score the strength of a hedge fund’s operations.

Canonical correlation analysis is most commonly used in epidemiological studies into things like the health effects of air pollution and the danger levels of other environmental factors. The process essentially entails searching for the highest relationships or correlations between potential causes and outcomes.

To perform the analysis on hedge funds, the academics first had to find reliable inputs for the model. They first tested the approach with data taken from Form ADV filings by hedge funds that were required to register with the US Securities Exchange Commission (SEC) in 2006. The analysis showed strong correlations between hedge fund failures, as recorded in the Lipper TASS hedge fund database, and the disclosures in Form ADV regarding past regulatory and legal problems, conflicts of interests, ownership structures and the use of leverage.

This allowed the researchers to weight 23 separate variables based on their correlation to fund failure and calculate a single score to represent a hedge fund’s estimated operational risk profile.

In 2011, the academics repeated the experiment with data from 444 ODD reports prepared by an independent service provider. This time they were able to identify 45 variables from their correlation analysis. This included instances of untruthful responses to questions posed by ODD specialists, which turned out to be the most important indicator of future problems. The results confirmed the findings of the earlier study: a higher omega score was indicative of greater risk of fund failure.

The omega score has met with a mixed reaction from industry practitioners. The paper receives praise for scientifically substantiating the correlations between operational risk factors and hedge fund failures.

For instance, the research provides an interesting perspective on the relationship between hedge fund leverage and operational risk. Although leverage is generally considered to be positively correlated with investment risk, with more highly levered funds expected to suffer larger losses, the opposite is true from an operational perspective. The study finds lower leverage corresponds with higher operational risk as prime brokers are less willing to finance hedge funds with poor reputations or weak processes.

The finding that factors such as regulatory problems, misrepresentations and ownership structure are leading indicators of operational failure is less surprising.

“[The paper] confirms many of the things ODD practitioners already know from experience,” says David Woodhouse, chief due diligence officer for the Jubilee Portfolio Management Group (formerly Fauchier Partners) within the Permal Group.

The main concern expressed by some ODD practitioners is that any all-in-one scoring system could mask a critical flaw in a manager’s operations if  they perform well in other areas.

“If a manager has a history of regulatory and legal problems, you don’t need to weight that, you need to fail them,” says one ODD specialist.

This argument is similar to the criticism levelled at VaR, which led many to ignore tail risks.

“The issues and tropes that we look for in ODD are too complex to distil into a single score in the same way the risks of a macro fund cannot be effectively captured by a metric like VaR,” says Woodhouse.

The Jubilee Portfolio Management Group has developed a sophisticated scoring system for ODD that covers around 100 variables across 56 operational functions.

The scoring system measures the strength of all these individual components but the ODD team ultimately comes to a qualitative view of the manager’s operations as a whole, rather than relying on the sum of the scores

“Any assessment of operational risk requires a nuanced interpretation of the underlying risk factors,” says Woodhouse.

“If you look at valuation, independence is clearly fundamental but there is often a tension between independence and expertise. There may be cases where an administrator or other third party does not have the ability to price certain assets and manager involvement, properly governed, will produce a fairer value. Similarly, net asset value restatement could be a sign of integrity rather than an indicator of fraud. To me it doesn’t make a lot of sense to score these factors in a binary way and build a universal model from that data.”

In his view ODD will also always require “a qualitative overlay”.

NYU’s Brown does not suggest that quantitative models such as the omega score can substitute for comprehensive ODD reviews and the need for qualitative insights into a manager’s operations. “I don’t think any type of formulaic approach will do the job,” he says.

Instead, he says the research can help ODD practitioners scientifically identify the areas where they ought to focus their attention. For instance, Brown found that hedge funds running convertible arbitrage, short bias, emerging markets and equity market neutral strategies had higher omega scores than the industry average, suggesting that these types of managers should be subjected to more rigorous ODD reviews.

At a more granular level, the results highlight the importance of performing background checks and comparing leverage relative to peers.

The lack of granular, uniform data remains the biggest hurdle to quantifying hedge fund operational risk effectively with statistical techniques. Brown reckons the analysis will become more precise and have wider practical application as hedge funds become more transparent about their operations.

Industry initiatives such as the open protocol enabling risk aggregation (Opera) could make this possible, especially if they are expanded to cover more operational areas.
Alternatively, this data could come from regulators.

“I’d like to get my hands on the Form PF data and start working with that,” says Brown. The SEC’s Form PF rule requires hedge funds to make detailed disclosures about their counterparty risk exposures, among other things, which has not historically been freely available to investors.

It remains to be seen if greater transparency and improved access to data will fuel the rise of the quants in the field of operational due diligence as it has in other areas of finance. Yet as things stand it appears that quantitative tools are no substitute for old-fashioned legwork and human judgment when it comes to rooting out fraud and operational problems at hedge funds.