Lloyds of London on Validation - testing and reporting enhancement ideas

Lloyds of London delivered a presentation around model validation last week to its 80-odd syndicates which anyone in the world of IMAP would benefit from picking through the bones of, bearing in mind the rather unique position of the Lloyds application (i.e. in the door of the PRA, and seemingly well received!).

They had noted in that presentation linked to above that they had found some weaknesses in 5 areas in particular, so this presentation is a deep-dive examining the strengths and weaknesses of validation - one may expect the other 4 'weak' areas flagged may receive similar treatment in coming weeks.

While their 2013 programme aims only to close the gap between full compliance with Solvency II tests and standards and today's position, it's worth flagging some fundamentals;

• Only half of syndicates felt to meet tests and standards in full - a third are 'pending' positive assessment, the rest have not passed.
• 'Fails' seem to be centred around following up on test failures and documenting findings in the summary report, rather than anything broader.

Around the production of Validation Reports, they noted negative findings around;

• Uncertainty about how to progress when something 'unacceptable' is found during validation testing
• Content of validation reports being statistic-heavy (i.e. indigestible to any non-quants who need to make decisions off the back of the findings)
• A lack of sophistication in the testing of material risks in some instances.

The last one is particularly interesting, as the central team at Lloyds has devised a schematic (slide 11) to show the kind of testing they expect to see on the more material risks (RST, P&L attribution) versus less material (going as far as qualitative tests).

It is also worth highlighting for any benchmarkers out there that Lloyds appear to advocate around 5 pages of Validation Report per risk factor, leaving their overall expectation of reports to be 30-40 pages, with 5-10 pages of appendices (p16). Bearing in mind these reports will I suspect be some of the first to go through the PRA's hands, the frame of reference may help encourage you to bulk up or slim down your own versions!

A large amount of this presentation (from p19 onwards) is devoted to fairly granular examples of how a validation test may be 'failed', and what action would be performed in order to gain a 'pass', so for those in the test design/conduct game, you may find something to support your approaches in that detail, regardless of the risks shown in the example (premium and reserve).