Losing track?

Tracking error is the gold standard of buy-side risk and performance measurement. But are investors short-changing themselves, trading more complex but insightful measures for this simple metric? Naomi Humphries reports

Tracking error – a measure of a fund’s performance relative to a benchmark – is the asset management industry’s standard gauge of portfolio risk and performance. But the market downturn since 2000 has jolted the investment community, and forced it to reassess how it measures the performance of its asset managers. Tracking error, the once unquestioned metric, is increasingly seen as only one of a handful of essential tools, including stress testing and risk and return decomposition analysis.

Tracking error’s prominence was boosted by the popularity of index-linked investing, which took off in the second half of the 1980s. By ‘managing to’ an index, asset managers sought to gain broad and representative exposure to an ever-rising market. For more than 10 years, the benchmarks did exceptionally well and the strategy was profitable, says Adam Seitchik, chief global strategist at Deutsche Asset Management in London. To be a good asset manager meant, in a sense, not falling behind the index. “Risk was defined in terms of under-performing your peer group, prompting the obsession with tracking error,” he says.

For index-tracking funds, maintaining a minimum tracking error was essential. Active (or ‘enhanced’) managers seek to ‘beat the benchmark’. For them, tracking error must be set to a reasonable level that reflects the chosen balance between risk and return in their investment strategy.

Ex post tracking error is a measure of how well the portfolio actually performed against its benchmark index over a certain time period. Fund managers also calculate ex ante (or ‘expected’) tracking error. Since the future returns on most portfolios and benchmarks are uncertain, this involves developing a probability distribution of returns for both the portfolio and the index, and comparing the two (see box).

“ Tracking error has an appeal in that it attempts to boil down all your risks into one metric, which is something that can be compared relative with actual performance and ex post tracking,” says Joe Marvan, head of the US fixed-income team at State Street Global Advisors in Boston.

While this seems straightforward, it has its drawbacks. For example, recent research by Goldman Sachs’ equity derivatives strategies team suggests the standard risk models do a good job in forecasting longer-term tracking error levels, but are poor predictors of short-term tracking error. This can deviate significantly from predicted levels, both on the upside and the downside. Some investment managers abandon tracking error altogether for very short-term portfolios, such as money-market funds, opting instead for value-at-risk or similar measures.

Ingrid Tierens, New York-based vice-president in equity derivatives strategy at Goldman Sachs, attributes the renewed interest in performance metrics to the ongoing stock market malaise. A few years ago, people were getting more tracking error in the short run than they anticipated because of low correlation across stocks and relatively high stock volatility. “Even though managers were aware of it at the time, it was not as much an issue to have performance variation around a benchmark that was up 20–30%,” she says. “But a bear market increases awareness – it’s not good to have more tracking error than expected when the market is already negative and clients are more risk-averse,” she says.

This means more accurate predictions become ever more important, which has caused managers and investors to modify their approach. When it comes to using forecast tracking error, Tierens says there has been a shift from a macro approach to a more bottom-up analysis, where managers are now focusing more on individual stock volatility and correlation, and connecting the two more explicitly. “The investment industry would benefit from a better understanding of how a metric like tracking error can behave differently in the short run as compared with the long run,” she says.

Consciousness of tracking error’s limits is beginning to spread. Although the ideal of one number that can reflect the risk-adjusted performance of a portfolio is very attractive, especially to less sophisticated trustees at public pension plans, to small corporate plans and among smaller asset management companies, the more sophisticated managers recognise that tracking error forecasts are in some cases fraught with problems. “Tracking error is a risk metric that we utilise, but it is over-utilised in some instances and by some clients,” Marvan says. “It is one model that offers one hint on some systematic risk, but it must be combined with other sophisticated models,” he says.

But tracking error has its value, asset managers say, as long as you keep its limitations in mind. “You have to take your historical analysis with a large grain of salt,” says CS Venkatakrishnan, head of JP Morgan Fleming’s US fixed-income quantitative research group in New York. “It is always a problem to calculate ex ante tracking error, regardless of whether your index is equities, fixed income, investment grade or below, but it is a particularly severe problem when indexes have been volatile,” he says. “Obviously, understanding past performance is important, but just as important, if not more so, is paying careful attention to the risk management systems, technology and processes managers have in place to measure ex ante or prospective risk.”

While tracking error does a reasonable job of measuring systematic risk, idiosyncratic or event risk is harder to model. Lou Gehring, New York-based senior vice-president and product manager at BondEdge, a fixed-income portfolio analytics system from FT Interactive Data, says he has seen growing interest from clients who want to look at the mismatch they have to the benchmark on an issuer-specific basis.

Indeed, “most portfolio managers now use a broad array of tools to augment tracking error”, says William Lloyd, London-based global head of portfolio strategies and index products at Barclays Capital. Increasingly sophisticated software, such as the systems offered by Barra, Askari and a host of other vendors, allows managers to drill down into portfolios and analyse the factors that drive their risk and return. This software is becoming more common.

Different measures of value are also gaining popularity, which has prompted BondEdge to provide valuations using RiskMetrics’ CreditGrades system, which uses a Merton-type model to measure companies’ credit quality. “Clients can look at the volatility of the credit grade on a name, compare it with the bond market valuation, then see how that relationship has changed over time,” says Gehring.

Barclays’ Lloyd advocates running scenario analyses and stress tests on fixed-income portfolios. The bank’s in-house portfolio analytics system, XQA, is designed to look at both expected and extreme conditions to generate a return profile, both absolute and relative. The exposures of a portfolio relative to its benchmark are laid out along several dimensions such as duration, term structure, rating, sector and issuer. Interest rate and credit spread scenarios are specifically designed for to perform a ‘what-if’ analysis on the portfolio and the benchmark.

Venkatakrishnan says credit portfolio management highlights some of tracking error’s shortcomings. “Part of the weakness of the tracking error approach is that it assumes that your performance within a sector would be similar to your index’s performance within a sector, that the index’s sectors were homogeneous and that you could reproduce them,” he says. “The reality is – especially in credit – the sector indexes are becoming increasingly difficult to reproduce because a large portion of the securities are illiquid and don’t trade.”

While investors and portfolio managers are eager to see how they perform relative to a benchmark, it is often difficult to choose an appropriate credit index for the sector in question. “This should be an entirely conscious decision, and because it may not be so in some sectors where the indexes are not ‘investable’, managers take that into account when looking at tracking error in the investment process,” Venkatakrishnan says.

JP Morgan Fleming supports the development of indexes that are more tradable and investable. However, there is still little consensus on how the new fixed-income indexes should be constructed, and how to achieve as much price transparency as possible. The launch of consortium indexes, such as the iBoxx products (see page 6), aims to overcome these problems by providing an independent index that is less subjective and more transparent than proprietary indexes, and focuses on more liquid sectors of the market.

Fergus Lynch, London-based managing director of index development at Deutsche Bank, and one of the creators of iBoxx, believes the new suite of indexes will fill a gaping hole in the fixed-income arena. IBoxx-based products such as Cristals – a corporate index tracker note that effectively gives investors the return of the entire index – are designed to appeal to both retail and also institutional investors with limited credit capabilities. “People want to be in the credit market in terms of risk return, but because of the recent credit blow-ups, everybody’s really nervous about name selection,” says Lynch. “They can buy a broad index replication instrument that gives them the index return, and nothing more, nothing less.”

But many still rely on the tried-and-true fixed-income indexes, which usually contain many credits. State Street’s Marvan says that, while indexes that are easy to replicate may be appealing, they may potentially be much more volatile. “If one name in the index defaults, it’s of much greater impact to the end client,” he says. “The best index is one that represents that broad market that offers you much greater diversification.”

Meanwhile, some question the wisdom of simply managing to benchmarks altogether. “The way we’ve been measuring risk – which is risk versus a benchmark – is actually a very small part of the risk of an institutional investor,” says Deutsche Asset Management’s Seitchik. He says the fundamental change in the investment environment in the past few years has placed benchmarks under review. With more than 20 years’ positive performance, the indexes became redefined in very optimistic ways. When the technology bubble burst, it became quickly evident that the indexes did not represent well-diversified portfolios. “It emerges that all the risk is in the benchmark itself,” Seitchik says. “We are just playing around at the edges to make a bit of alpha, but actually it is very irrelevant to the risk and return of the client’s portfolio.”

Indeed, the really important tracking error may not be relative to indexes at all, but rather between assets and liabilities, says Paul Bostock, London-based managing director at asset management firm GMO. “We are currently encouraged to offer returns close to the index, good or bad, rather than good returns. An alternative might be moving over to performance measurement according to realreturns, such as absolute targets for fixed liabilities,” Bostock says.

Calculating tracking error

Calculating ex ante tracking error – essentially, forecasting the deviation of a portfolio from its benchmark – runs into similar issues to calculating a portfolio’s value-at-risk. Any of the three standard techniques for calculating VAR – variance-covariance, historical simulation and Monte Carlo – can also be used to develop a portfolio return distribution to calculate tracking error. However, while financial institutions often calculate daily VAR for trading desks at a –1.65 standard deviation level (a 95% confidence level) or higher, asset managers have to predict return performance further into the future. Also, while VAR measures downside risk, tracking error measures both upside and downside risk. For portfolios that include many individual securities, risk model providers such as Barra and Northfield reduce the dimensionality of the risk calculation by casting it in terms of a limited number of underlying factors that are supposed to drive the security returns. Unfortunately, the problems that plague the three techniques for producing a probability distribution of returns for VAR also affect ex ante tracking error calculations. The weaknesses generally lie in the assumptions required. The variance-covariance model relies on historical data, and makes the assumption of a normal distribution of outcomes. The historical simulation approach also relies on past data, and assumes that a portfolio’s future return distribution will be similar to its historical distribution. Monte Carlo requires assumptions about portfolio performance to set the parameters for the simulations. All three calculations assume a static portfolio and a static index. Changes to an index’s composition – a fairly common occurrence for both equity and fixed-income indexes – can lead to significant changes in its risk. Also, the volatilities and correlations of both the portfolio and the index may be unstable over time. The tracking error value may change, not because the relative exposure to risk factors has changed, but because the volatilities of or correlations among the risk factors have drifted. Conditions such as high volatility, shocks or political and economic changes can all devalue the usefulness of using past performance as an indicator of future returns. Changes in the underlying risk factors may not follow a normal distribution – the true distribution of returns may be skewed.