Model validation is a key element for internal model approval under Solvency II, but it is one of the most demanding. Clive Davidson reports on how insurers are meeting the challenge One of the key requirements for regulatory approval of an internal model under Solvency II is that it is properly validated by the insurer. The primary reason is to ensure that the level of regulatory capital held by the insurer is not miscalculated to the extent that it undermines policyholder protection. But it is not just a matter of regulatory compliance. No model will predict the future with absolute certainty and, given the central role that internal models will play in the business decision-making and risk management of insurers under the new regime, it is essential that insurers take all the necessary steps to ensure their model is as accurate, robust and appropriate as possible. Model validation aims to do just that. According to the European Insurance and Occupational Pensions Authority (Eiopa): “Validation is a set of tools and processes used by the undertaking to gain confidence over the results, design, workings and other processes within the internal model.” (Advice for Level 2 Implementing Measures on Solvency II: Articles 120 to 126 Tests and Standards for Internal Model Approval). These tools and processes must be quantitative as well as qualitative “as it is highly unlikely that the undertaking will be able to design quantitative processes that will be able to explain sufficiently all the results and processes of the model”, according to Eiopa. Developing and implementing the tools and processes presents a number of challenges, not least because model validation under Solvency II requires a far more detailed and formalised process than most insurers are likely to have had in place, if they previously had a formal validation process, or indeed a model, in place at all. The requirements of Solvency II have prompted Amsterdam-based ING Insurance to create a new validation function for its internal model, headed by Mark Vermeule. “Under the Solvency I programme, there was no pressure to produce internal models, and therefore no necessity for model validation,” says Vermeule. However, with Solvency II following the lead of Basel II for banking and introducing the option of internal models for regulatory capital calculation, coupled with the heavy criticism the Basel II models received following the 2008 banking crisis, the need for a proper model validation process in Solvency II is fully justified, Vermeule adds. Insurers have no choice but to devise a validation programme if they are seeking internal model approval, says Gerald Suggett, programme manager for Solvency II at Royal London Group, but putting it in place is by no means straightforward. “The [model validation] requirements [of the Solvency II Directive] are very broad and firms need to be able to interpret and convert them into a sufficiently rigorous validation framework,” he says. Ann Muldoon, Solvency II director at Friends Life, based in London, agrees. “During the [model] implementation phase there is a need to ensure a consistency of what validation means – the language used in the Solvency II legislation can mean different things to different people,” she says. One way to get a grip on the validation process, which can mirror the model itself in its complexity, is to divide the model into a number of smaller components so that it becomes more manageable, suggests David Wong, London-based partner at global consultancy PricewaterhouseCoopers (PwC), which is advising a number of insurers on their validation programmes. “It is also important to build validation around a framework that clearly demonstrates how validation activities fit into the architecture of the model,” says Wong. He recommends preparing flow charts of the model processes in which validation points or model controls are highlighted and then clearly described in an associated matrix. “This can help ensure that the right level of validation is carried out and also makes sure that the validation activities are transparent and clearly explicable to the regulator or board, as well as helping to articulate the ongoing cycle of validation once the model has been approved,” he says. Model validation covers a number of key areas, all of which have specific regulatory requirements set against them. They include data quality, validation methods, documentation, systems and other IT, model governance and the use test. External models, such as economic scenario generators and catastrophe models, are not exempt and insurers “must ensure that external models and data are consistent with the standards and requirements set out for the use of an internal model to calculate the Solvency Capital Requirement”, says Eiopa in its Level 2 advice. Two areas that demand particular attention in the validation process are assumptions and expert judgement. “Experience of the banking crisis highlights the vital importance of focusing on the assumptions and expert judgements that underpin the capital evaluations, rather than the model engineering, as these are likely to have the greatest influence on the validity of the numbers and the management actions they help to inform,” says Wong. Insurers have to address how to provide “a trail of auditable thought processes or discussion behind expert judgement decisions”, says Royal London’s Suggett. Insurers will want to make use of a number of tools in their validation processes, including back-testing (testing the results against past experience), sensitivity analysis, stress testing and scenario analysis (testing the robustness of the model including areas where the model breaks down), profit and loss attribution (testing the completeness and appropriateness of the model) and analysis of change (why have results changed over time). Benchmarking, or peer review, is another tool insurers may want to use in time when appropriate data on industry models becomes available. Experience with Basel II and in other industries where models are commonly used suggests that there can be a number of pitfalls in the validation of models. Insurers are trying to take this experience on board in devising their own validation programmes. Many agree that the most important pitfall to avoid is to treat validation simply as a compliance task. “To validate only to align with minimum regulatory requirements is short-sighted,” says ING’s Vermeule. “As a best practice, a model validator must pose the ultimate question: is this a good model?” Under Solvency II, a model is ultimately a senior management tool for business decision-making and risk management, therefore the question of whether the model is fit for this purpose is as important as whether it aligns with minimum regulatory requirements, he says. Suggett agrees that validation should be “a rigorous independent challenge of the internal model” and not just an internal checking process. Insurers must avoid “passively accepting the results of the validation process and not using the process to generate upward pressure on the internal model change process,” he says. Key to validation being an effective challenge of a model is ensuring the results in the validation reports that the model produces are intelligible to those whose job it is to ask the critical questions. This raises the question of who exactly should be involved in the validation process. At Friends Life, the risk management function ‘owns’ the validation process and is responsible for ensuring the completeness and independence of the overall process and outputs, says Muldoon. The risk management team has developed framework standards and a set of recommended tests for the various business areas in the firm to carry out. “We have a three line of defence approach [to validation],” says Muldoon. “In the first line are the business units, which are responsible for completing the tests. The second line of defence is the risk function, which sets the strategy and policy, provides an oversight to the process and challenges the scope and effectiveness of validation. “The third line of defence lies with group internal audit, which provides independent assurance that the validation process is appropriate and is carried out as intended,” she says. Ultimately, model validation must become an integral part of the regular business and reporting processes, adds Muldoon. Royal London recommends a similar three line of defence approach. In addition, it suggests that the board or board risk committee undertake a number of functions, including identifying and reviewing major validation activities, identifying any shortcomings to the firm’s validation approach, identifying materially unsatisfactory results from the model, and recommending improvements to the firm’s validation policy. Finally, it should sign off the appropriateness of the validation processes used. “Under no circumstances can this role be delegated,” says Suggett. As part of its governance framework for its internal model, ING Insurance has a model committee, which is a forum for discussing model developments and validation reports. The governance framework includes a statement of the model life-cycle, identifies all the relevant stakeholders, and sets out the validation policy, which is based on Eiopa’s Level 2 advice and describes the types of validation to be performed and the requirements to be met. ING Insurance distinguishes between pre-approval and periodical validations. “Whenever a new model is built, we perform an extensive pre-approval validation. After that, a periodical validation is performed on such a model, which can be light or extensive depending on the performance of the model,” says Vermeule. In addition, the validation team can also issue what are called ‘no objection statements’. “Sometimes early in the model development phase, the developers ask the model validation team to take a look at the work thus far and say if they think the direction they are heading in is the right one,” says Vermeule. If the validation team agree that the developers are at least not heading in the wrong direction, they can issue a no-objection statement. “This is not a formal validation, because it’s just too early in the process to analyse the outcome of the model,” says Vermeule. The actual validation process at ING Insurance consists of two main parts, says Vermeule. “First, we check the minimum requirements with respect to documentation, data quality, and so forth. We also check the assumptions and test the conceptual soundness and performance of the model. Second, we take a step back and try to answer the question: is this a good model?” The validation team discusses the outcome of both parts and writes a draft report. “After that, we have a discussion with the model developers, before we submit our final findings to the model committee,” he says. Given the critical importance of model validation, it is essential to hire people with the right qualities for roles in the validation process. Candidates must not only have appropriate expert knowledge and intellectual qualities, but also the skills to communicate with all the stakeholders in the model, as well as the “integrity and strength of character to be able to challenge as an internal critic,” says Vermeule. In addition, the validators must be organisationally separate and independent from those developing and running the model. “Not separating validation activity from development and running the model and so failing to show true independence is a pitfall to be avoided,” says Suggett. On the other hand, it is important not to over-engineer the validation process to the point where it is no longer agile and cannot meet the changing demands of the business, says Friends Life’s Muldoon. Model validation is one of the six key tests for internal model approval under Solvency II and in many ways the most important, “as it makes sure all the other [tests] are working properly”, says PwC’s Wong. But it is also one of the most challenging. In a report on progress being made by companies seeking model approval in February 2011, the UK’s Financial Services Authority said: “The validation policies that we observed during our review fell below the standard that the Directive requires.” (Solvency II: Internal Model Approval Process, Thematic review findings). Since then many companies, such as Friends Life and Royal London in the UK and ING Insurance in the Netherlands, have made good progress with their validation framework and processes. But all companies will have to make validation a priority if they want internal model approval and for their model to be a benefit to their business....
Start a FREE trial or subscribe to continue reading:
Start a 4 week free trial
Try Risk.net's premium content for a limited period. Register now for your FREE trial to one of our leading brands.
*not available to previous trialists or subscribers.
Log In or Subscribe Now
Subscribe to Risk.net now to access our premium content for 1 year.
Pay by Credit Card for immediate access.