PRA on "good" and "bad" documentation for IMAP - restart or refresh?

As an welcome aside from the PRA's activity day on the 12th, they also delivered on a promise to provide examples of what constitutes "good" and "bad" documentation in the context of the internal model approval process (IMAP). Bearing in mind it has been quite some time since the PRA publicly pronounced on the matter - indeed, they were called something else the last time they did - the industry welcomed this in like a lottery-winning prodigal son.

Bad documentation
- reform or execute?

There have certainly been enough apocryphal tales spewed out by the industry over the last couple of years to suggest that the production of documentation to support participation in IMAP had transformed from a small cottage industry into something of a palatial Georgian Mansion house with a granny annex.

Whether or not this kind of advice is therefore timely enough to save the Solvency II Programmes of the UK from using documentation in their revitalised IMAP preparations that is neither good for man nor beast is another thing, bearing in mind there is three years worth of accumulated flotsam currently kicking around the servers of UK insurers which, given this message, will need to be revisited as a matter of urgency.

Regardless, as the UK still appear to hold the whip hand in EIOPA's model sub-committee, the PRA's views on model documentation are relevant to readers across the continent, so the document is well worth a read for anyone in pre-application.

I took the following from it;

• They have 3 principles of "good documentation" - accessibility, evidence and quality control
• "It is helpful to have a clear separation of policy, methodology and results"
• The document refers throughout to giving consideration to "the reader". Feels a little disingenuous if they mean "the PRA", and after reading their document a couple of times, I can't imagine who else they have in mind.
• Following on from that point, a reference to a bad example suggests that "...the author is not thinking carefully about the audience" - it is equally fair to suggest that the author may be assuming a level of technical/commercial knowledge at the PRA end which is lacking?
• The PRA are evidently not happy to have to "seek clarification" on matters, which suggests the time allocated to assessments is tightly planned, or indeed the level of technical knowledge held by the assessors is limited.
• Interested to know if these principles hold firm for Standard Formula firms if viewed conversely e.g. does paperwork for the justification to not use an internal model need to be of a similar standard?
• The reiteration that the PRA "...will rely, in large part, on the submitted documents" when assessing the model - just in case your friendly executive committee reckon they can talk their way in!
• A lofty aim to improve "accessibility" of documentation, by having levels covering; executive summaries; model reviewer/validator level info; and model user/operator level technical documentation. The PRA only want the first two levels as a matter of course, which would suggest that procedures and technical documentation should be used as supporting evidence only.
• Tabling up references to the Directive and Implementing Measures within documents is viewed favourably.
• Their "useful rule of thumb" feels instinctively unwieldy - they suggest better documents contain one-third 'what has been done' with the rest covering 'why and how'. That said, expert judgements are clearly not being evidenced anywhere near the level desired.
• A convoluted and potentially frightening reference to "self-validation" testing, which the PRA view favourably, but which seems to point towards a preference to see suites of documentary evidence for each assumption applied in the modelling process. 
• To conclude, readers are redirected to Julian Adams's letters from  mid- 2012 (here and here) - important to note that this guidance remains relevant, regardless of the passing of time.

I suspect that most model applicants will find they have re-work to do off the back of this, both with the pen and with the sickle. Expert judgement and assumptions documentation remain the unquenchable thirst though, so one's best endeavours would be well spent in that field in early 2014.