Risk experts Eric Talley and Johan Walden of the University of California Berkeley, who reviewed the stress tests on behalf of the Congressional Oversight Panel, pointed out that "there was some criticism that both sets of assumptions were too optimistic at the time, and there was additional criticism when the economy deteriorated further after the... exercise began". Specifically, they noted, real unemployment is already above the level predicted in the more adverse of the two stress scenarios: compared with a forecast average of 8.9% for 2009, unemployment had already hit 9.4% by May.
In the first quarter of 2009, the economy contracted an annualised 5.7%. The adverse scenario included a contraction of just 2% in the whole of 2009, but forecasts from the International Monetary Fund in April predicted a 2.8% contraction this year, and 0% growth in 2010, compared with the adverse scenario's 0.5%. If this trend continues, the report recommended, the Treasury should repeat the stress tests with "more difficult economic assumptions". The tests should in any case be repeated "so long as an appreciable amount of toxic assets remain on their books, economic conditions do not substantially improve, or both," the report added.
A key mistake highlighted by the report was the Treasury's failure to be more open about the methods as well as the results of the stress tests. Though two scenarios were tested, only one set of results – those from the more adverse scenario, used to calculate capital requirements – was released. Releasing the baseline scenario results as well would allow outside analysts to extrapolate the sensitivity of the banks to other scenarios. "The results of the stress tests under the 'baseline' economic scenario should be released or Treasury should explain why they were not released," the report said. In particular, different mortgage portfolios were assigned varying predicted loss rates, based on no clear criteria: "An element of judgement was necessary in determining these loss rates. It allowed the testing, for example, to reflect local conditions with greater accuracy. However, because of the judgement involved, the calculations cannot be reviewed or replicated. This diminishes the reliability of the tests and the confidence that the public is able to place in them."
The Treasury was also far too prone to trust banks' own descriptions of their conditions, the report said. "Significant interaction required between supervisors and the [banks] has the potential of undermining the objectivity of the stress tests," the panel warned. It added that it was unclear whether supervisors had double-checked banks' figures: "This is not to question the good faith of either the supervisors or the tested institutions. But the experience of the past two years cannot but cause some to question the adequacy both of the risk management practices of many of the nation's largest financial institutions and of the scope of the supervisory regime to which those institutions were subjected."
A third flaw was to limit the test to a short period – the scenario only included losses up to the end of 2010. But this meant the stresses did not include possible delayed effects of the crisis. "A recent study commissioned by Deutsche Bank suggests that the majority of losses on CRE [commercial real estate] loans will not affect bank balance sheets for several more years when poorly underwritten CRE loans made in the easy credit years (for example, 2005–2007) will reach maturity and will in many instances fail to qualify for refinancing... The extent to which the stress tests, which were never intended to look more than two or three years in the future, fully grapple with the prospect of massive future CRE loan defaults is uncertain". By contrast, UK regulators recently announced their intention to test banks against a four-year-long stress scenario.
See also: Treasury silent on Tarp as banks submit $75bn SCAP capital plans
FSA stress tests banks for four-year downturn
Stress-test success masks bigger problem with banks
US banks require $74.6 billion in additional capital, stress tests reveal