IRM on Internal Model Validation - Red Card or Green Card?

Cyclic Validation - Quelle horreur...

Just back from Paris, where I spent a weekend queueing behind selfie-taking tourists before taking out a second mortgage to buy bottled water. A beautiful place, though I found Depardieu was much quieter off-screen...

Onto the topic in hand, the PRA were pretty vicious back in the day on Validation efforts in their infancy, with Julian Adams lambasting both progress ("significantly behind") and validation scope ("narrow"). Given that the Solvency II sabbatical which bridged half of 2012 and all of 2013 gave firms time to catch up and widen, you might think that those with internal model ambitions would be pretty tidy by now. The PRA have even told firms how they believe "good" model application paperwork to look, carving out for themselves and the Validators of the world an easy-to-read "model reviewer" level of detail (p1).

In those salad days, Internal Model Validation felt to me like it would be the chernozem of the nascent Risk Management profession in insurers; a skill set that a quant or a non-quant could acquire, apply, and ultimately ease through the promotional path within insurance entities, given the depth and breadth of technical and strategic information the process challenges...

...but the moves never came. Despite the actuarial world themselves happily disassembling the complexities of quantitative modelling into easy-to-digest IM Validation themes, the non-quant world has waited patiently to see if anything of substance would emerge from one of its representative bodies.

And this week it arrived! The Institute of Risk Management has delivered, as part of its Internal Model Industry Forum (IMIF), a white paper on the validation cycle.

The IRM have been active in this area prior to the formation of the IMIF. I have covered an ERM in Insurance event at the start of 2014 here, while this more volumous slide pack featuring a number of the Billy and Betty Big Biscuits of the field emerged from summer of last year, when the IMIF seemed to come to fruition. This white paper itself appears to move along the concepts and ideas inside an IRM slide deck from last Christmas.

Given that the IRM is not-for-profit, there is always a likelihood that sponsors will unduly influence the products (indeed the IRM Chair notes in this that they rely on "enlightened industry support" to knock these documents out).

Sadly in this case, the sponsors include Three of the "Big 4" (with the fourth on the IMIF steering committee) , leaving the document dripping with consultancy hallmarks rather than pragmatic solutions to execute the tasks in hand.

That view is reinforced somewhat by this follow-on presentation to the IMIF from last week by this white paper's workstream lead and supporting consultant - one selected industry comment on slide 8 (presumably from a chocolate bar shortly before it ate itself) reads, "validators should really be experienced modellers"!

A few general points jump out of the white paper;

• That a firm's IM is "...at the heart of risk and capital evaluation" - I thought it was supposed to "inform" this evaluation, not dominate it (slide 3 here, as well as Julian Adams's speech from a couple of years ago [p4]).
• Is the insurance industry "...increasingly reliant on sophisticated models" - maybe in terms of AUM/Market Cap, but given the UK IMAP queue is down to approximately 40 firms out of over 400 (p4), and that number has steadily reduced over the last 3 years, feels a touch disingenuous. I've no doubt the firms represented on the Steering Group are "...increasingly reliant" though
• The document claims to set out "best practice principles" - not sure if "practice" and "principle" share the same bed, but that aside, would anyone find it remotely acceptable to have the consultancy world fund a document which details "best practice" on IM Validation?

And a few stand out elements from the proposed Validation Cycle, which is heavily influenced by EIOPA's guidelines:

• "Best practice now requires firms to demonstrate, with evidence, that the cycle...[is] being actively and effectively carried out" - how can best practice "require" anything from anyone?
• "...resulting best practice that is emerging" (p4)  - how is any practice considered "best" at this stage of proceedings, when we are literally practising! Against what criteria?
• References to "model risk impact assessment" and the "model risk assessment process" (p5) seem to come from nowhere. Alluding to something formal, but not very clear
• Lot of coverage of "triggers" of IM Validation, which feels like a fishing expedition for the paper sponsors, rather than direct address of L2 Art 241 - the number of areas of "change" to consider as IM Validation triggers covers pretty much any change, anywhere, both inside and outside of an insurer (p8)! Most would also be ad-hoc ORSA triggers in my experience, so this potentially sets up insurers for a bucketload of work every time they hear a pin drop.
• Formulaic and periodic IM Validation a "needless cost"? Surely periodic validation, no matter how badly executed, is compulsory (L1 Art 125)?
• The Trigger Impact Assessment stage (p10) is barely legible - "The trigger impact assessment against model risk appetite stage" - and terminologically it is all well above legislative requirements.
• "Unexpected triggers" (p12) get a mention. Again, not making sense to me - you either know your triggers or not.
• "Model validation is complex" and "less than black and white" (p16) - certainly is if you try and follow this process! A focus on plain questions and less quant can only help the models non-expert users (slide 7).
• If the validation cycle, processes and execution are "continuously evolving" (p18), are they reliable? Feels difficult to meet L2 Art 241.3, at least from a planning and execution perspective, if the process is constantly being tinkered with 
• "Developing a communications strategy" (p20) as part of the validation scoping and planning stage feels terribly over-elaborate.
• "Robust planning" expected to be common (p22), which doesn't necessarily marry up with the expectation of dynamic rather than cyclic validation in future (p10)

I think it is right to take the hump to a certain extent here. The PRA have been cunningly silent on capital add-ons to date, but given the implication that they will not be applied and renewed ICG-style (slide 13), there is likely to be many more less monied Partial IM applicants to follow over the next couple of years. Having the most influential consultancy firms decide on what is "best" in the validation world (and for it to have this many bells, whistles and legislative off-roads) feels like setting those firms up for either a fall, or another bill.

The PRA actually delivered something with much less padding to the IRM back at the end of 2013, so I'm struggling to see why that has justifiably been turbo-charged. Given they have three of their finest involved with the IMIF, but are continuing to be directly vocal on this topic (as recently as March 2015), it sends a worrying message to the capital add-on brigade that the IMAP early birds will be setting disproportionately high bars for 2017 and beyond when they deliver their PIMs.

Ultimately, I was disappointed by the publication, which reads more like a flannel manual, and is certainly not the kind of Risk Profession contribution that the topic so badly needs if the PRA's dreams of Board's "directing" and "owning" the IM valdiation process (slide 9) are ever going to come true. The 200 page novella world of Validation Reporting feels closer than ever...

Jurassic Talk - enhanced NED challenges during Solvency II preparations?

 Britain's youngest NED

Given that there won’t be a heck of a lot more briefing done on the Non-Executive Director front, I’ve given the PRA Industry Event slide pack published the other week a bit more of a going over to see if the left and right hands are pushing the Solvency II wheelie bin in the right direction. I haven’t gone as far as watching the 1h 30m video of the event yet – if I wanted to watch a room full of fidgeting old men in ill-fitting suits I’d just go to Bridge Night at the bowling club…

I can’t say I was massively enthused by the read of the slides as an individual who is frequently delivering material to the very audience the presentation was aimed at. I would highlight the following oddities;

Internal Model-specific (slides 6-12)

• That Solvency II “sets a high bar” for model approval – that feels a little disingenuous given that the PRA has had the whip hand in the IRSG’s internal model committee for years, and has evidently driven their CAT. Fair to say that the PRA set the "high bar" on behalf of the rest of Europe.
• From the “lessons learned” section they suggest that some IMAP/CAT firms have used assumptions which are not matching their experience. Is that not bravado bordering on criminality? Doesn’t feel like small beer, so unless the PRA are splitting hairs with that comment, I trust the protagonists had a strip torn from them.
• Some models ignoring “Key Risks” faced by a firm – how can this be? If this is about cheeky risk selection (i.e. let’s use SF, but model Market Risk as we get a good number from it) all well and good, but to say that firms are ignoring them is not a good steer, and if they are, then how is punishment not already being dispensed?
• For Use Test purposes, NEDS told to have “belief” but not “blind faith” – this feels like Bank Creep, given that the PRA  have been vocal (here and here) on firms blindly following models after the banks got caught with their trolleys down a few years back (nice PRA summary here). Doesn’t feel especially fair to tar insurers with exactly the same brush in advance, even if it is smart!
• “Boards need to own validation design” – just sounds meaningless when you read it back. If you want them to “do” the design (which Andrew Marshall’s later slides deriding the efforts of the validation contracting community suggest also support), then just say it.
• The “Key Questions” slides contain some very ropey gear. “Does the output of the model give a credible answer”? “Can the firm survive on the Standard Formula”? The terms used are so flimsy that one could spend hours arguing the toss about their definition – “so what is survival – EC+, SCR+, MCR+ with recovery plan” etc.

ORSA and SoG (slides 14-18)
Starts with a bit of good news – some generic industry feedback is seemingly due within the next couple of months (pertaining to our 2014 ORSA efforts?). The slide summarising findings to date is also a useful yardstick for those who can’t wait that long.

For System of Governance, the executive world should prepare themselves for NED questions regarding whether or not they (as opposed to their underlings and contractors) are reading EIOPA's Guidelines. Let's hope they have!

On the gnarlier side;

• Seems to be an obsession with assigning named individuals (as opposed to roles or teams)  to perform mitigating tasks relating to anything ropey uncovered during the ORSA
• “ORSA should be holistic” – at what point is that breathtakingly grim term going to be put to pasture? For a NED briefing, the use of plain English should be considered par for the course. It is followed two slides later by “top-down/bottom-up” which is equally non-specific.
• “ORSA is not a compliance exercise resulting in a report to the PRA” – I think you meant to say “not ONLY...”!

The final slides from Ian Marshall’s presentation are revealing more due to the clumsy terminology often used at the table with NEDs (“Key Drivers” and “Key Correlations” for example – if you mean “most money riding on it”, then say it!). Also, the idea that Risk Appetite is “no longer an aspiration” is worrying – I would have given the insurance industry credit that it ceased to be aspirational some time ago, and doesn’t need a 2015 ‘tick’, but then I am a trusting fellow.

Does anyone think, off the back of these slides, that their NEDs will be chomping at the bit at the next Risk Committee/Board meeting using the ammunition supplied here?

Maybe I’d better watch the video after all… 

EIOPA's Implementing Technical Standards on Internal Model Approval - ready to "submit"?

EIOPA have kicked into life at the end of October like Freddy Krueger on laughing gas, releasing a swarm of consultation papers and summaries. While the releases on Colleges of Supervisors or the calculation of the risk-free interest rate will be of interest to some, my own interest was in a subset of their draft Implementing Technical Standards on Solvency II Approval Processes

I have tried to cover the distinction between ITS and the other weaponry in the hands of EIOPA and legislators here. The aggregated feedback received by EIOPA on the suite of Solvency II approvals covered by ITS is neatly summarised in this doc, with EIOPA adding dollops of proportionality into the summary, to effectively remind supervisors not to treat the applications of giants and pygmys with the same vigour. Given the audience reaction to the UK regulator's industry event a couple of weeks ago, I think 'proportionality' may need to become the PRA's middle name (though it probably works better as its first name...)

Clearly, two big moans have been common threads in the responses received, and both have been rebuffed. The concept of supervisory requests for supplementary information "stopping the clock" on one's approval window has been retained, while the idea of "no answer in 6 months equals approval" has been firmly rejected.

Internal Model approval
- "Easy, Easy"

Of more interest to me than the other approvals covered by the consultation is the Big Daddy himself, Internal Model Approvals. From a UK perspective I have covered this in multiple posts previously (here, here, here and here for a start) , and was interested to see if any more whistles and bells had been added to what is already an area causing serious fatigue within the UK's potential applicants, not to mention a few dozen corpses from the c.100 interested parties back in 2011 (slide 4 here).

I thought the following was worth highlighting;

• Given that the Delegated Acts are already loaded with granular Documentation requirements for IM applications (articles 243 & 244), the ITS insists on a few more pieces! Having cross-referenced between the DAs and the ITS, I reckon the items listed in Article 3 d, h, j, p, q and r are all new requirements, though none would be a stretch to produce.
• Expectations that your model has been in use prior to application (Art 2 (3 a ii)), and is integrated into your current system of governance (Act 2 (c)) - perhaps this is the effort which firms with capital add-ons will be compelled to do in 2016 & 2017 in advance of model applications?
• Results of your "last" Validation Process must be submitted in a Model application (Art 2 (m)) - again, suggests there will need to be a year's worth of "live" modelling in advance of making an application
• The supervisors have 30 days to assess the "completeness" of an application - this feels weirdly generous, given those 30 days count towards the overall 6-month assessment window (provided the application is complete).
• Despite the PRA's firm assertion on Oct 17th (slide 8) that conditional approval for Model applications is not available, Articles 3 (8) and 5 (4 b) both suggest otherwise, namely that proposed "adjustments" or "terms and conditions" can be accounted for in their decision. 
• As much prominence being given to Model Change process and Model Change policy as the initial Model Application process - applications can actually be rejected on the strength of the change policy (Art 5 (2)). Despite that, the articles designated for these areas (Arts 7 & 8) are relatively straightforward.

There is no real reason why these ITS for IM approval won't sail through at the Commission, so best to prepare accordingly. However, UK applicants with 2016 modelling ambitions will of course need to wait further for EIOPA's Common Application Package before they can finalise this montagne de papier.

PRA's Countdown to Solvency II Implementation Event - When I Need U(SPs)

PRA Implementation countdown
- "Feel like dancing" now?

So along with the entire UK Solvency II sticky-beak brigade, I had the unbridled pleasure of attending the PRA's 'Leo Sayer' on Friday, confidently titled Countdown to Implementation, despite going on to list a range of matters which have probably given both Internal Model and Standard Formula firms plenty of enthusiasm to try and turn the clock backwards rapidly!

Paul Fisher, Julian Adams' replacement as Executive Director of Insurance at the PRA, kicked off with a set of Solvency II vox-pops;

• Solvency II is the "main game in town"
• "The end is in sight"
• The PRA are "not gold-plating the Directive"

A nice bit of reassurance at kick-off time then - unfortunately, the clarity which was to follow from the technical specialists on a range of topics was probably not what everyone wanted to hear, given the tone of some of the audience questioning that followed! In particular (and with everything in full quotation marks below having been said by a PRA rep on the day);

• That the PRA will "have to prioritise" if everyone in the IMAP queue for 'from 2016' approval drops their applications in at the end of June, and that applicants should be "striving to submit" sooner. Suggests to me that the smaller firms in IMAP may get bumped to squeeze in the big boys.
• The reinforcement of Mark Carney's message from the other week that they will have no problem refusing permission to use models, though this was couched by Fisher in the more appropriate context of failure to meet any of the TSIMs simply "cannot be allowed".
• That, although over 90% of the UK industry was down for using Standard Formula, the PRA will be equally aggressive when challenging their preparedness as they will for IM firms.

I didn't hang around for the afternoon sessions as I had a hot date with BA, so pull whatever you can out of the Other Approvals and Regulatory Reporting slideshows. However, I would draw attention to the following from the earlier sessions:

Implementation and Policy overview

• The Old Lady of Threadneedle Street protested through multiple speakers about how they recognise that bank and insurance internal models are different, and how Insurance Supervision is now "embedded into the Bank" - I would assume to justify the credit crunch-influenced aggression now being taken by the PRA on assessing capital models (visible from Adams's speech around the time of BOE/FSA merger right through to Bailey's speech at Mansion House on Thursday night).
• An interesting slide 3 about how much more work Solvency II will generate for the PRA from go-live!
• Referred to FLAOR only once before switching to ORSA, suggesting that this disposable acronym is as much of a pain in the neck for the supervisor as it is for firms!
• Highlight what "Good" and "Bad" ORSAs have contained to-date. Clearly some firms are box-ticking, leaving an unusable report which is only skin-deep compliant as output. They were particularly scathing on Stress and Scenario Testing efforts, and implored that Reports should not be written for the PRA's benefit (though naturally must cover what the DAs and EIOPA have already set out).
• A nice piece was discussed around the expected depth of director-level knowledge of their internal models. Andrew Bulley made a useful distinction between "conceptual" and "technical" knowledge, where your dithering 80 year-old INED from the fishing industry might not be expected to understand correlation matrices, but should probably know their significance, and alternatives to them.
• For model applications to date, far too big ("encyclopedic" in cases), with too much process description, and not enough on assumptions and expert judgements.
• That it is a firm's responsibility to "ensure compliance" with the Delegated Acts, and given their lessening proximity to the legislators, the PRA flag in advance that they will not be able to give "concrete advice" to firms in future.

Standard Formula

A particularly good ground-setter, given the dearth of work published previously on Standard Formula firms and the PRA's expectations. Calendar included on the slide pack, which will be of massive use to your PM/PMO staff!

• PRA will review ALL firms ahead of 2016 for SF appropriateness - "priority" firms assessed by Q1 2015, everyone else by end of 2015.
• While SF SCR is apparently close to current ICG numbers for GENERAL insurers in the UK, it is noticeably larger in LIFE firms 
• PRA is "not promoting" SF ahead of internal models
• Vigorously directed all attendees to EIOPA's Underlying Assumptions of SF Paper - expectation that some firms won't read it, and just expect SF plus any add-ons?
• Very vocal on the "significance of the deviation" between SF SCR and one's own Risk Profile - as we know, the Delegated Acts quantify what 'significant' is in the context of capital add-ons
• An expectation that ORSAs will be used in the assessment of SF appropriateness - qualitative for sure, possibly quant elements as well?
• Range of examples of where significant divergences are being found, by Risk Category, and by Life vs General Insurer.
• A lot of emphasis on Capital Add-ons being used "only as a temporary measure", which will ultimately allow firms to PIM/IM or de-risk. They are however "patient and realistic" on how quickly that change can be done, so it sounds on the face of it like 2016 and 2017 will be targeted for Capital Add-on elimination.
• In the following session on models, a piece came up on Capital Add-ons, where the PRA confirmed that their process for handling these in future is "still developing", though they expect them to be "a lot, lot rarer" than the existing regime of ICG.

Internal Models

Calendar also provided for IM firms (slide 4), showing how tight their schedule is, and explaining why they threw the earlier curveball about all firms expecting to drop their applications in on 30th June 2015. They also touched on the following:

• Highlighting weak areas identified to-date such as over-optimistic (new?) business plans being used in capital calculations; ENIDs; omission of certain "Key Risks", and suspiciously low correlations
• That Use Test is "fundamentally important", and is an opportunity for firms to "put their money where their mouth is". They do not expect to see either end of the use spectrum i.e. no use, or blind use!
• Too much technical actuarial validation seen. Usefully suggested that validation questions may be better posed as "where might this model be inadequate", rather than "why is it OK".
• Confirmed that the PRA's SAT has now been replaced by EIOPA's CAT, which won't arrive until the back end of this year - surprisingly, no-one laughed when they said this "might create some work" for existing IMAP programmes!
• Importantly, they stressed that their powers are to Approve or Reject applications, with no "conditional" powers whatsoever. Attendees were therefore encouraged to delay applications which were thought to be unlikely to succeed, both now and in future.

Though they instinctively feel like they could have been supplied sooner, the clarifications provided in the presentations I witnesses will be hugely welcome by programme directors and PM's alike. I would venture a guess that they will be less welcome by executive committees, who may have hoped for more flexibility on the risk quantification front post-2016.

Solvency II Delegated Acts available online (kind of...), plus EIOPA's plans for 2014/15

So let's start with something a bit unexpected - DRAFT DELEGATED ACTS! ONLINE!

I'm not sure who the leaky uploader is (appears to be a Spanish consultancy firm), but the document is very much online. Sadly, it is only the January 2014 version, which as you will see in the rest of the post, has just been superseded, but it definitely pairs up with the version currently doing the rounds, I promise!

I have managed to get a sneak preview of the latest version of this document (dated 14th March) which have seemingly managed to burst the banks of the tightly-knit circle of advisors, and are now no doubt winging their way to a Solvency II Programme Director near you! There are "tracked changes" on the March document now circulating, which only appears to cover changes since the emergence of the January document hyperlinked above.

Lord help anyone who wants to trace it back to the more familiar 2011 (unpublished) draft, you might as well draw a load of foxheads on sticks...

Insurance Europe were obviously part of the privileged few for the March revisions, hence they fired out this missive last week regarding all of the Pillar 1 technical areas which they feel (on behalf of the industry) remain deficient. There are no real surprises in their list - it is the same topics which have been on the whinge-list since EIOPA's LTGA last year, and indeed earlier in the case of the Currency Risk approach and Own Funds classification.

Following on from the draft Delegated Acts being made more widely available, there has been a reasonable amount of noise in the paid-for press (here, here and here for subscribers), as well as Insurance Europe's top man having a lobbying call published in the FT (here).

Being more of a Pillar 2 man myself, I thought I would check to see what, if anything, had been tweaked in my areas of interest. The impression given earlier this year was that little had changed outside of the Long-Term Guarantee elements, and that was certainly true if you compared the November 2011 and January 2014 documents.

However, having examined the amendments in the March 2014 version, I have found is that a few areas of governance (both SOG and Internal Model governance) which were previously untouched have actually received a fair bit of treatment, for example;

• Changes to the requirements for internal audit function holders not to cover multiple control functions (this constraint has been removed). This is presumably to pacify the smaller firms across Europe who have a Risk/Compliance/Internal Audit multi-tasker, so textbook "three lines of defence" have taken a bath in the interests of proportionality.
• The devil remains in the detail though, as the amended text allows someone to "carry out" more than the IA function, but seems to stop shy of them "taking responsibility" for other functions. Not sure how that will work in practice.
• Changes in the IM Validation space, in particular the removal of the requirement for a "Validation Policy". Fair to say most firms in IMAP would have produced one of these at least a year ago now (plenty of industry references here, here, here (p8) and here for example!), and while still a document of merit, does a "validation policy" now constitute gold-plating?
• Changes in the required Internal Model Documentation, targeting a much slimmer set of compulsory documents. This includes replacing a number of "policies" with "descriptions of...", which will no doubt be well received by those supervisors with multiple internal models to assess over the next 18 months!
• The tiered timescales for submitting QRTs, SFCRs and RSRs have now moved into the Directive, via Omnibus II text (as opposed to haveing been deleted, which is what it looks like at first glance!)
• A few of the other TSIM articles (Tests and Standards for Internal Model Approval) have been enhanced. "At least quarterly..." assessment of the IMs coverage of material risks is now specified, for example. Quite how the hard-coding of the regularity cramps your actuaries' style is another thing! 

I strongly suggest you all get back to work and check for yourselves!

Model Validation - benchmarks and best practices from the IRM, PRA and Lloyds

Models - seeking validation

Bearing in mind we have passed the 2013 financial year end, and all of the internal modellers amongst us will be trying out their new processes and technology in the production of a balance sheet for the first time, a few very timely pieces of material on internal model validation have been released, which should help anyone who is curious about what their neighbours are up to!

The Institute of Risk Management's rebadged Solvency II Special Interest Group held a get-together in December on IM Validation, and a couple of interesting documents have emerged.

One from a PRA Validation guru is particularly useful for anyone in the IMAP space who has any uncertainty about the PRA's approach to assessing the quality of one's validation processes and reports, covering;

• The purpose of validation
• The PRA's approach - Life and GI-specific SME groups
• An overview of IMAP findings - 40+ reviews conducted to date, and they also state what your evidence "should" demonstrate
• Their observations - Noting that Validation Reports are generally deficient
• A schematic view of what they consider the Validation process to be
• A bullet point list of how validation effectiveness can be demonstrated

I guess the odd thing for me is why this kind of material isn't presented/circulated more widely by the PRA, as it is surely of benefit to IMAP participants, who wouldn't always be in attendance at a pre-Christmas IRM event!

The other useful benchmarking item from this event is the survey on participants' experiences to date in the validation field. While the sample is small in absolute terms at 18, and is a touch heavy on the GI side (over half of respondents), this is as good a benchmarking aid as you will see for a while, so it is worth noting the following;

• Just over half have transitioned Validation into BAU
• Only 3 respondents had a Validation-specific steering committee to help govern the process, with others choosing to use existing committees or the CRO/Risk function
• Over 80% use the SCR contribution of each risk driver to determine the depth of validation activity. Other determinants include regulator feedback, risk registers and previosu validation reports
• Over 40% say that their independent validation work is identifying flaws in their "dependent" validation work, while over a quarter say that independent validation has been scaled back due to the quality of "dependent" work!
• Less than a quarter say that validation of external models (ESGs etc) has been effective
• Around two-thirds use peer review and sensitivity analysis to validate expert judgements. Horrifically, two respondents said they haven't been able to validate expert judgement at all!
• Over half are still using external contractors/consultants for independent validation
• Page 12 covers the popularity of certain sections of a Validation Report. Less than half include a section on benchmarking
• Most are keeping the Validation Report to under 100 pages, with management feedback being the main catalyst for changing the length.

Finally, a survey from LCP Consulting was published this week covering the progress of Lloyds syndicates in their Validation activity. This was an area which Lloyds acknowledged as "work to do" back in May last year, particularly around validation testing and documentation.

The findings appear to be positive on the whole, with most firms saying they are at least halfway towards their "ideal process". Unsurprisingly, dependency modelling and validating expert judgement make the list of "key challenges" remaining.

PRA on "good" and "bad" documentation for IMAP - restart or refresh?

As an welcome aside from the PRA's activity day on the 12th, they also delivered on a promise to provide examples of what constitutes "good" and "bad" documentation in the context of the internal model approval process (IMAP). Bearing in mind it has been quite some time since the PRA publicly pronounced on the matter - indeed, they were called something else the last time they did - the industry welcomed this in like a lottery-winning prodigal son.

Bad documentation
- reform or execute?

There have certainly been enough apocryphal tales spewed out by the industry over the last couple of years to suggest that the production of documentation to support participation in IMAP had transformed from a small cottage industry into something of a palatial Georgian Mansion house with a granny annex.

Whether or not this kind of advice is therefore timely enough to save the Solvency II Programmes of the UK from using documentation in their revitalised IMAP preparations that is neither good for man nor beast is another thing, bearing in mind there is three years worth of accumulated flotsam currently kicking around the servers of UK insurers which, given this message, will need to be revisited as a matter of urgency.

Regardless, as the UK still appear to hold the whip hand in EIOPA's model sub-committee, the PRA's views on model documentation are relevant to readers across the continent, so the document is well worth a read for anyone in pre-application.

I took the following from it;

• They have 3 principles of "good documentation" - accessibility, evidence and quality control
• "It is helpful to have a clear separation of policy, methodology and results"
• The document refers throughout to giving consideration to "the reader". Feels a little disingenuous if they mean "the PRA", and after reading their document a couple of times, I can't imagine who else they have in mind.
• Following on from that point, a reference to a bad example suggests that "...the author is not thinking carefully about the audience" - it is equally fair to suggest that the author may be assuming a level of technical/commercial knowledge at the PRA end which is lacking?
• The PRA are evidently not happy to have to "seek clarification" on matters, which suggests the time allocated to assessments is tightly planned, or indeed the level of technical knowledge held by the assessors is limited.
• Interested to know if these principles hold firm for Standard Formula firms if viewed conversely e.g. does paperwork for the justification to not use an internal model need to be of a similar standard?
• The reiteration that the PRA "...will rely, in large part, on the submitted documents" when assessing the model - just in case your friendly executive committee reckon they can talk their way in!
• A lofty aim to improve "accessibility" of documentation, by having levels covering; executive summaries; model reviewer/validator level info; and model user/operator level technical documentation. The PRA only want the first two levels as a matter of course, which would suggest that procedures and technical documentation should be used as supporting evidence only.
• Tabling up references to the Directive and Implementing Measures within documents is viewed favourably.
• Their "useful rule of thumb" feels instinctively unwieldy - they suggest better documents contain one-third 'what has been done' with the rest covering 'why and how'. That said, expert judgements are clearly not being evidenced anywhere near the level desired.
• A convoluted and potentially frightening reference to "self-validation" testing, which the PRA view favourably, but which seems to point towards a preference to see suites of documentary evidence for each assumption applied in the modelling process. 
• To conclude, readers are redirected to Julian Adams's letters from  mid- 2012 (here and here) - important to note that this guidance remains relevant, regardless of the passing of time.

I suspect that most model applicants will find they have re-work to do off the back of this, both with the pen and with the sickle. Expert judgement and assumptions documentation remain the unquenchable thirst though, so one's best endeavours would be well spent in that field in early 2014.

Pre-application for Internal Models - EIOPA's FINAL preparatory guidance for national supervisors

So the March consultation document for internal model pre-application brought a few eye-openers for those countries partaking in a less onerous application process than that favoured by the UK, with the detail in it suggesting that the UK very much had the whip hand in its drafting.

On the basis that there were still areas which even the most hardened IMAP-veteran may have winced at, it was interesting to see if anything got dropped in the lobbying stampede. On that basis, the final guidance for internal model pre-application covers the following in the preamble;

• 3.10 - That it is not in the NCAs gift to conduct pre-application preparation along the lines of provisional approval or to provide "roadmaps" to compliance (which may explain the PRA's caginess with the industry). It is purely about a firm's preparedness and suitability to submit an application
• 3.33 - On request, EIOPA have introduced a compulsion for NCAs to provide "regular feedback" to firms
• 3.35 - Confirms that not all model changes need to be reported to NCAs during pre-application, just those considered "relevant" by firms themselves

It is fair to say that the lobbying in this space has been noticeably more successful than for ORSA or System of Governance, no doubt due to the smaller sub-set of affected stakeholders having a more concentrated relevance. That said, there were still a number of rebuffs from EIOPA, particularly where the lobbying looked more like whinging about paperwork volumes! Highlights below;

Model Change Policy

3.38 - No danger of EIOPA supporting the recommendation to "fast track" model change approvals if a "major change" is required at short notice. They instead recommend "proactivity" with NCAs. Not sure what this does for the world of opportunistic acquisitions though

3.40 - Fudged the question as to whether parameter changes are considered "major", offering an answer of 'it depends' which, for me at least, leans more towards 'yes they are'.

Use Test

3.42 - Confirms that evidencing "use" is not compelling use of model outputs over and above other techniques

Assumptions and Expert Judgement

3.46 - Documentation and validation of assumption setting and expert judgements considered "crucial" in order for undertakings to counter the lack of data and subjectivity in those processes

3.47 - A guideline has also been amended to confirm that the materiality principle applies for this topic

3.48 - Only the most material assumptions will need AMSB sign-off

Methodological Consistency/PDF/Calibration

A number of changes made to clarify guidance in these areas

Profit and Loss Attribution

3.64 - No escaping the requirements to produce P&L attribution granularity at Legal Entity level, as well as by risk driver

Validation

3.68 - EIOPA do not accept that those who build models may also validate them

Documentation

3.70 - That the guidance around the documentation of the internal model should provide "...[protection] from key-person risk", which I have never seen offered as justification from the supervisory end before

Ultimately, there have been no huge concessions from the position in March, which one would think will cause a number of the more liberal EU regulators to give serious consideration to "explaining" rather than "complying" - that said, with this having been written in Union Jack ink, my British cousins should simply get their transition planning updated accordingly.

Lloyds of London on Validation - testing and reporting enhancement ideas

Lloyds of London delivered a presentation around model validation last week to its 80-odd syndicates which anyone in the world of IMAP would benefit from picking through the bones of, bearing in mind the rather unique position of the Lloyds application (i.e. in the door of the PRA, and seemingly well received!).

They had noted in that presentation linked to above that they had found some weaknesses in 5 areas in particular, so this presentation is a deep-dive examining the strengths and weaknesses of validation - one may expect the other 4 'weak' areas flagged may receive similar treatment in coming weeks.

While their 2013 programme aims only to close the gap between full compliance with Solvency II tests and standards and today's position, it's worth flagging some fundamentals;

• Only half of syndicates felt to meet tests and standards in full - a third are 'pending' positive assessment, the rest have not passed.
• 'Fails' seem to be centred around following up on test failures and documenting findings in the summary report, rather than anything broader.

Around the production of Validation Reports, they noted negative findings around;

• Uncertainty about how to progress when something 'unacceptable' is found during validation testing
• Content of validation reports being statistic-heavy (i.e. indigestible to any non-quants who need to make decisions off the back of the findings)
• A lack of sophistication in the testing of material risks in some instances.

The last one is particularly interesting, as the central team at Lloyds has devised a schematic (slide 11) to show the kind of testing they expect to see on the more material risks (RST, P&L attribution) versus less material (going as far as qualitative tests).

It is also worth highlighting for any benchmarkers out there that Lloyds appear to advocate around 5 pages of Validation Report per risk factor, leaving their overall expectation of reports to be 30-40 pages, with 5-10 pages of appendices (p16). Bearing in mind these reports will I suspect be some of the first to go through the PRA's hands, the frame of reference may help encourage you to bulk up or slim down your own versions!

A large amount of this presentation (from p19 onwards) is devoted to fairly granular examples of how a validation test may be 'failed', and what action would be performed in order to gain a 'pass', so for those in the test design/conduct game, you may find something to support your approaches in that detail, regardless of the risks shown in the example (premium and reserve).

Internal Model Validation - the "desire for certainty"

Some useful snippets on the links here for anyone in the model validation space, whether if be the practical applications of Monte-Carlo simulation outside of the insurance industry, actuarial perspectives on model risks themselves such as parameter uncertainty and goodness of fit testing where "the problem is more often too many candidate distributions" as opposed to restrictions in choice.

This fantastic blog post from one of Willis's finest is about as blunt a critique of actuarial modelling activity and its potential for subsequent misuse as I have read, and I would strongly recommend it on to non-expert risk practitioners who may one day find themselves in the model validation/use test firing line. A few of the pearls of wisdom offered (focused on reinsurance industry, but relevant to all) include;

• How the human "want to believe" and "desire for certainty" can lead to models making rather than guiding decisions
• Reliance of models on "large numbers of heroic assumptions"
• "Data is always limited and flawed"
• That "models take combinations of assumptions and torture them to come to conclusions"
• The revisiting of assumptions only when the answers don't fit expectations ("euphemistically called 'calibration'", hilarious!)
• That using models for setting regulatory capital, rather than just informing decision making, has led to "extremely onerous" IMAP activity i.e. the limitations noted above are so well established that the regulators cannot ignore them at a granular level.

Then there this piece from Deloitte US on model validation, or more specifically, research into the quality of existing actuarial modelling controls, is an eye-opener for anyone working in the validation space. With RMORSA and associated capital modelling firmly on the agenda Stateside, it is interesting to watch how aggressively they approach validation, bearing in mind this work was commissioned by the Society of Actuaries, whose members may ultimately be charged with applying some of these recommendations!

This research in particular assesses current state versus best practice controls over the assumptions, inputs and outputs of actuarial models, and though the sample of respondents to the survey is relatively small (representing "30 unique companies"), the absence of suitable supporting documentation around model governance so evident in the UK's IMAP process appears to be a depressingly constant theme. This report at least includes recommendations as to how the US actuarial profession may bridge some of the gaps Deloitte identify.

In the NAIC's ORSA Manual, they ask that "ORSA Summary Report should provide a general description of the insurer’s process for model validation, including factors considered and model calibration" (p7), which I guess is what one expects to see in the EU (i.e. validation being a sub-process of the ORSA, which can be summarised in the ORSA reports). That said, the breadth of validation work performed over there will surely be driven by S&P expectations communicated in ERM Level III reviews, rather than profession-sponsored consultancy recommendations!


Finally (and slightly off track), an odd piece from Towers Watson on validating ORSAs, pitched to a room full of Internal Auditors. Would be unfair to say there aren't some salient points throughout, but given that there is "no clear requirement" to validate ORSAs (there was something on the matter in the original CEIOPS ORSA pre-consultation, but it was dropped in the public consultation and the final advice), then you would think it could be covered in less than 30+ slides!

As it happens, the TW slide pack for internal model validation appears to have been raided and had the acronym 'ORSA' jemmied into the text for much of the second half of it.

Lloyds directors briefing and Solvency II update - looking well

With Solvency II news and comment pretty thin on the ground this month as the guys at EIOPA count their LTGA beans, it was nice to see the trailblazers at Lloyds release their director briefing slides from last week, an event one may assume is relatively frosty after the money spent in preparation and the distinctly agitated tone on the matter from their CEO recently!

Bling - things Lloyds could have
bought with £300m

As Lloyds are already well down the road of internal model development (having almost kept to their original IMAP deadline they were able to get materials down to the FSA prior to the implementation deadline shifting), the slides are very revealing as to where the group remain lacking when attempting to meet the Tests and Standards for Internal Model approval (TSIMs). With 84% of the market 'by materiality' meeting the principles of the TSIMs, they are clearly in good nick, although by stressing 'materiality', it implies that a relatively large number of smaller members are perhaps still lagging.

The list of "common issues" found will neither surprise nor delight anyone else in IMAP, given that the same themes have been festering for a good 18 months now, and the legislative paralysis on the continent has clearly done nothing to aid the industry (in particular, the FSA letter from this time last year touches on most of these!). Specifically, they observe problems in the following areas;

• ORSA - looking far enough forward (i.e. past year 1), and using stress and scenario testing effectively
• Validation - evidencing validation work done, and following up on test failures
• Model Change - justifying the thresholds for minor/major changes, and agreeing an approach for aggregating minor changes so that they can be considered as minor/major in aggregate
• Use Test - using the model for something other than spewing out an SCR, and it would appear also that when interviewed, the effectiveness of board training and their understanding of the model is being found wanting
• Documentation - documents are either not checking off against the TSIMs, or the content is contrary to the revised controls and processes which have been developed for Solvency II

As Julian Adams made clear at the turn of the year, full compliance with TSIMs is not part of the ICAS+ agenda down at the PRA, however they will expect firms to be fully aware of where they are currently light, and what they plan to do about it. Certainly looks like the Lloyds application won't struggle in this regard, and I wish them well.

The PRA's take on EIOPA's Interim Guidelines - far from a 'Tragedy'...

The Prudential Regulatory Authority have celebrated their second week in office by hosting a number of industry briefings regarding EIOPA's Solvency II Preparation Guidelines. One Blogger keenly dripped some of the materials discussed out yesterday, but I've had a look through the briefing materials released today on the PRA's site to see if there are any messages worth amplifying.

Steps - appropriate?

From a calendar perspective, the PRA are planning some technical workshops with "industry representatives" in early May, to be followed by an industry briefing later in that month. The output from the workshops appears to be a major influencer on whether the PRA will "comply" with EIOPA's guidance or "explain" why they won't.

With final versions of EIOPA's guidelines expected by September/October 2013, the national regulators have an additional 2 months from publication to formally confirm their "comply or explain" position.

Reading between the lines this looks (for now at least) as a foregone conclusion however, with this comment from the internal model pre-application slides;

"We expect firms to have regards to the interim guidelines and take appropriate steps to prepare for Solvency II" 

Three slide sets have been released - anything new, or worth reiterating, below;

Internal Model Pre-Application

• Model change - to be monitored throughout pre-application.
• Colleges - Evidently notable divergences in assessment practice across the union which the Guidelines hope to fix, but "no direct implication for firms".
• IMAP - while EIOPA's guidelines "will be considered as part of IMAP reviews", they don't (in the PRA's mind) touch on anything which wasn't already part of existing L1/Draft L2/pre-application L3, so no surprises on that front. They do however highlight the elements on Expert Judgement and Validation as providing more clarity on supervisory expectations.

ORSA and System of Governance (combined)

• No PRA expectation of  a full/'live' ORSA or Solvency II-compliant systems of governance by 2014
• Neither are there any expectations for firms to make changes in investment strategy/capital strategy/outstanding Pillar 1 elements - at least not due to Solvency II in its current form.
• For System of Governance, the PRA's slides highlight areas where EIOPA's guidelines exceed expectations of the PRA Handbook - these include Board MI; revisions to control functions; revisions to Risk Management Policy; Prudent Person Principle; and calculation of TPs.
• For ORSA, the slides clearly indicate that dry runs and ORSA process development should be the immediate focus, and it should serve to reaffirm best practices that already "underpin the ICAS".

Submission of information to supervisors

• Only a subset of the full package recommended by EIOPA in July 2012
• EIOPA want to see at least one round of annual submissions, and two rounds of quarterly submissions before Solvency II goes live, with 2016 seemingly the preferred year.
• On that basis, firms will have 20 weeks (26 weeks for groups) after year-end 2014 to submit annual template obligations.
• They will then have 8 weeks (14 weeks for groups) after quarter end Sept 2015 to submit their quarterly template obligations.
• As per ORSA, firms which are involved with internal model pre-application will need to perform both standard formula and internal model work in this area.
• Also as per ORSA, thresholds for participation based on TPs (Life firms)/Premiums (Non-Life).
• Narrative reporting requirements are "significantly reduced"compared to final requirements - however, there is plenty of crossover between what one might document during ORSA processes and what is expected to appear in RSR/SFCR, so firms shouldn't struggle in this regard.
• The concept of a "Reporting Policy" document (guideline 33 in the consultation) covering who will be responsible for what in the submission of information to supervisors, is referenced in these slides. This hadn't featured on my radar before as 'compulsory', so documenting this early, even just as process steps/maps, seems like a move that the PRA would appreciate.

EIOPA preparatory guidelines - pre-application for Internal Models

Consultation on Pre-application for internal models guidance  (plus explanatory text)

As staggering as it is frightening, and perhaps indicative of the diversity of approaches currently on parade across the Union, the pre-application for IMs preparatory guidance is 60 pages, comprising of 72 guidelines, accompanied by a whopping 144 pages of explanatory text. This accompanies the existing 82 pages of L3 guidance on the matter released by then-CEIOPS in 2010! A sub-group of EIOPA's IRSG has been assigned to deal with the nitty gritty of this element of the preparatory guidelines.

On first read, this feels massively influenced by the UK's activities to date, and indeed anyone working in that space will recognise FSA pawprints all over the granular details contained within. This is fair I suppose - the InsuranceERM models map has the UK down for around 1/3rd of models currently in 'pre-application' across the continent.

Therefore bearing in mind the UK approach is already pretty well established, I have highlighted below areas which either diverge from what is currently being exercised on the ground, or which clarify (at least for me!) areas which were previously ripe for controversy or disproportionate/inconsistent application. Where it is common sense or continuez tout doit, I have ignored it.

Most importantly, I am reading it as a fait accompli - bearing in mind the short window of time between consultation end and period commencement, EIOPA's recent past on consultation responses (i.e. 'thanks but no thanks') and the UK's evident participation in the bulking of these guidelines, I don't see much room for lobbying swathes of this away, nor for the PRA to "explain" rather than "comply"!

GENERAL GUIDELINES

Guideline 3

• As well as nature, scale and complexity, "design, scope and qualitative aspects" of the IM should be considered when allowing for proportionality 

Guideline 4

• Any model changes pre-application look like they will be pored over by NCAs, including the associated change approval process 

MODEL CHANGES

Guideline 5 - Model Change Policy

• Policy should, as well as SCR-related changes, include changes to: system of governance (around model change); compliance with Use Test requirements; appropriateness of technical specifications and changes in Risk Profile

Guideline 6

• Approach to classifying "major" changes is expected to be objective
• Must also take into account specificities of the company (so benchmarking percentage changes against your neighbours may not be that useful) 

Guideline 7

• Aggregated change triggers must be considered, not just isolated changes
• Offsetting positives and negatives won't be acceptable to avoid a "major change" trigger!

Guideline 8

• Major/minor changes must be determined at Group and entity level

USE TEST

Guideline 9

• "No complete and detailed list of specific [model] uses" will be supplied by NCAs.

Guideline 11

• Granularity of the Risk Management System will need to match the IM in terms of categorisation
• The "structure of decision making fora" will be assessed in ensuring the IM fits to the business - extraordinary!
• Records expected to be available to show how IM outputs are designed

Guideline 12

• Assessment of training, seminars, workshops, meetings and direct interviews "should be considered" in pre-application

Guideline 13

• Will need to "ensure [IM] will be used" during pre-application, as opposed to "use it"
• Expectation that, if other tools are used in decision making, IM is improved having assessed inconsistencies against said tools

Guideline 14

• Evidence of prospective support and retrospective verification of decision making would be advisable for candidates

Guideline 15

• Must document where model is not aligned to the decision ultimately made

ASSUMPTION SETTING/EXPERT JUDGEMENT

Guideline 19

• "Materiality" in the context of assumptions will need to be both qualitatively and quantitatively assessed - should generate some healthy Risk/Actuarial function debate!

Guideline 20

• A validated and documented process for assumption setting and expert judgement will be required
• Sign-off on assumptions will need "sufficient seniority", up to and including AMSB

Guideline 21

• A formal and documented feedback should be maintained between assumption setters and users

Guideline 22

• On the transparency of assumption setting, point 1.64 here effectively asks for an Assumptions Register, as well as dictating what it expects to see in it.

Guideline 23

• Process mapping of some kind expected for the validation of assumption setting
• Independent assumption review is also expected - doesn't dictate whether this should be internal/external, but it will keep someone in clover no doubt.

METHODOLOGICAL CONSISTENCY

Guideline 26

• Methodological consistency to be validated

P&L ATTRIBUTION

Guideline 37

• Point 1.105 seems to confirm P&L attribution by risk driver is required

Guideline 39

• P&L attribution must be used at least annually in the decision making process

Guideline 40

• P&L attribution to be used in the validation process (specifically, old ones to be compared against  experience

VALIDATION

Guideline 41 - Validation Policy to contain at least

• Process, methods and tools, and their purposes
• Frequency of validation for each part of the IM, and triggers for ad-hoc validation
• Persons (not roles) responsible for each task
• Procedure to be followed where reliability of IM is questioned, and ensuing decision making process

Guideline 42

• Shies away from touching Internal Model scope when talking of validation scope - great move!

Guideline 43

• Evidence of sensitivity testing expected when determining materiality

Guideline 44

• Must document known limitations of validation process, as well as circumstances where the process falls over
• May even be asked to quantify the degree of uncertainty!

Guideline 45

• A documented escalation path would be advised

Guideline 46

• Risk Management function will be pressured, as the function with overall responsibility, to ensure all tasks are completed (if not directly performing them) - new skill set?

Guideline 47

• Evidence of how the RM function ensures that the validation process remains independent of IM design and ops should be collected/enhanced

Guideline 49

• A process will be expected to ensure the choice of validation tools used considers; complexity, nature, independence and knowledge of participants - feel this could be tricky for the smaller IMAP guys without leading to additional spend on consultants

Guideline 50

• Must be able to document the appropriateness of the validation tools used accounting for; materiality of IM part, granularity of the data being tested, purpose of the task and the expected outcome

DOCUMENTATION

Guideline 53

• Expectation of a "...clear referencing system [for IM documentation] which should be used in a documentation inventory"

Guideline 55

• An overall summary of IM shortcomings, "consolidated into a single document" will be expected
• This needs to cover at least; Risks not modelled, limitations in modelling, sources of uncertainty in results, data deficiencies, external models/data, IT limitations and governance limitations.

Guideline 56

• Potential suggestion that there should be more than one level of IM documentation to suit other audiences/uses - IM for Dummies anyone?

Guideline 57

• End-to-end User Manual expected which an Independent Knowledgeable Third Party could operate

Guideline 58

• Stress that a single document containing all model outputs (as Use Test evidence) is not required - acceptable as single docs

EXTERNAL MODELS AND DATA

Guideline 60

• Expectation that external data sets will be sense-checked against "other relevant sources"

Guideline 61

• Understanding of external models must extend to technical and operational aspects, as well as assumptions

Guideline 64

• "Material" assumptions of external models must be validated
• In point 1.163, any potential for cherry-picking features/options of external models is constrained

FSA on ongoing appropriateness of internal models (which aren't appropriate yet...)

Ploughing on regardless like a John Deere with a lobotomised driver, the FSA continue to work on their plans for ongoing appropriateness of internal models after Solvency II goes live. Having put their initial ideas out for feedback in June, they have this week provided an update on responses received, which hinted at a few things;

• IMAP Participant apathy - 10 responses (attrition rate is potentially rising these days, but we must still have 60-odd with skin in the IMAP game, so that feels pretty lousy)
• That inappropriateness would only be to a firm's benefit, hence the supervisory response to its detection "in all but exceptional cases" will be a capital add-on (PS if 'inappropriateness' is a word, I'll mange my chapeau, but I'll stick with it for now).
• That the early warning indicators planned will form part of the FSA's BAU Supervisory Review Process alongside "in particular" model validation results - any danger the early warning indicators may therefore be used informally in the pre-application work? They do go on to stress in the letter that they "do not intend" to use early warning indicators in the initial approval process, but bearing in mind no-one showed up for round 2 of the three-way today, we're all eating at a pretty moveable feast right now!

The link between early warning indicators and validation results is probably the big message in here - could verging on breaching the % tolerance, plus a negative validation report, lead to a capital add-on in 2014/2015/20XX?

KPMG on Solvency II - Progress in an uncertain world?

Handy survey from KPMG pushed around today touching on technical practices of UK Life firms (35 responded to the survey, so a decent enough sample). Appreciating the irony of the title in the same week as the rumour mill on an extension of Solvency II go-live to 2015 went into overdrive, the content is still valid on the benchmarking front, so worth a comb through.

Of particular interest I noted;

• IMAP remains top priority for those in the application process, though KPMG highlight that firms are closely reviewing the advantages of the IM versus standard formula due to FSA fussiness, implementation costs snowballing and less-than-attractive results coming out of the model (i.e the fabled 20% reduction guess-timates are perhaps out of reach!). Some comfort there for anyone struggling with the same issues.
• Documentation second priority for IMAP and non-IMAP firms - concerning at this stage, bearing in mind the FSA will have had plenty of opportunity to feed back on the big ticket items. If the industry still doesn't have a handle on regulatory requirements (or it does, but is not meeting them!), then perhaps it's a good job there is an extra year in the offing! 87% of the IMAP respondents said IM documentation specifically is causing challenges.
• Pillar 3 also high on priority lists - might we expect that to slip back on the agenda with the extra year? Particularly around software solutions, which may have previously been the only way for some laggards to populate QRTs etc within the proposed timeframes, some may decide to do something a little more ad-hoc for 2013 rather than full dry-running. Two-thirds of respondents note that turnaround time is a concern, and unsurprisingly 'achieving sign off' is a concern of around half, with the tight timescales naturally impacting on the likelihood of getting an executive to confidently put pen to paper.
• Only a third of respondents have dry-run their ORSAs - feels instinctively light, with those yet to run spread across the next 5 quarters. Natural areas of concern are later highlighted in this area, with ability (or lack of) to project across the business planning period, and plugging the ORSA process into the BAU strategic decision making process troubling around half of respondents. Bearing in mind only a third of respondents have performed any on-to-one training (with group training/workshops seemingly the preferred route), they may find that one will facilitate the other!

Some other noteworthy points;

• Only 80% think Solvency II will "significantly" impact the wider business - struggle to see how it could not frankly, but appreciate their are non-IMAP firms in the mix here
• Disturbingly large number of IMAP candidates appear to be behind the curve on P&L Attribution and Validation - interesting to know on the latter whether that is in respect of their internal findings or the FSA's view.
• Two-thirds of IMAP candidates using external assistance for validation - wonder how long that gravy train stays on the tracks with a shift to 2015?
• 20% don't have TPs in their IM scope - is that a bit chancey at this stage? I thought the FSA view on scope was a bit more onerous than some applicants may wish for.
• Datawarehousing seemingly moving towards being the preferred route to meet Pillar 3 obligations where an existing solution has not already been implemented prior to Solvency II

Good intelligence for both IMAP and Non-IMAP firms, and should provide some programme managers with crumbs of comfort when they see what's eating their counterparts right now.

Lloyds of London on Validation - Workshop Output

For all of you working on one of the FSA's (and indeed I suspect other supervisors') pet peeves, Model Validation, the Lloyds guys published some slides from their recent workshop on the topic.

Anyone not in the IMAP space may still find their presentation of interest, particularly as they have already had to ask for 3 months leeway on their original application "landing window", perhaps hinting at the enormous complexity and expense of producing material to support a successful model application.

They start with a "Top 5" ways to improve existing validation practices, rather worryingly stressing coverage and ranking, which one would think would be par for the course rather than something which required additional emphasis. They go on to hint at, if not confirm;

• That both analysis of test results and subsequent escalation has been found to be lacking during their validation dry run
• That historical syndicate experience is "always necessary" in modelling, even if not sufficient on its won
• An over-reliance on sensitivity testing using pre-defined SCR ranges
• Their recommended materiality hierarchy of validation tests
• Their take on reserving risk (not my forte, but dig in if it's yours)
• Their take on correlations and the diversification matrix (interesting bit on diversification in the tail)
• That their model walkthrough exercises brought up material issues for "most" agents, and that clearly some syndicates have been asked to complete remedial action before the end of this month
• That they are waiting for L3 before reviewing their validation reporting formally.

Not sure whether this keeps the guys on track for their group application, but certainly not an unhealthy thing to be picking up such flaws at this stage - if it was that easy, everyone would be doing it!

FSA Speech - Adams at Insurance Day Summit on Internal Model Progress (or lack of...)

Another day, another bum-kicking for the IMAP candidate firms, who have been the recipients of some public words of 'encouragement' from the FSA's Julian Adams in a speech to the Insurance Day Summit yesterday, coupled with a "Dear Firm" letter which went to all internal model applicants this week.

The same themes which have pitched up in recent presentations (here and here for example) are still for all to see - model change, model scope, validation and expert judgement/rationale all feature heavily in the speech - but perhaps the elaboration on both assumption/dependency setting and documenting and Use Test embedding are breakout features from those already tabled as being insufficient.

Useful for those outside of the UK to see exactly what is twisting the FSA's tail at the moment, while for all you UK readers, the letter is clearly a laundry list which will require immediate attention and prioritisation in your work plans. Good luck...

PwC paper - Solvency II Pillar II "operational issues of risk management"

Following on from KMPG's efforts, the PwC crew have released their own aide-memoire for Pillar II specialists, which at the very least should be handy for CRO/Head of Risk etc amongst us for board training/briefing ideas as we approach the home straight.

At 60 pages, it's not exactly a comic, and while a great deal of the introductory material is 'Pillar 2 101', it does contain material (either text or schematics) which should be of immediate use, so I would recommend checking out;

• First line of defence and reasons for risk ownership (p26)
• Definitions of Risk Appetite/Risk Tolerance/Risk Profile/Risk Limits/Risk Budget (p35) - I haven't cross referenced these against IRM definitions for example, but may be worth doing for consistency. An accompanying schematic is on p36.
• Internal Model Scope (p40) and Validation (p48) should be extremely topical after Mr Adams's speech on Thursday.
• Communications and training plan ideas (p53-54)
• ORSA schematic, and some talk around process industrialisation (p56)

Perhaps the main weakness in the text is that there are no references to the Commission's draft Level 2 text, which is now clearly a document in a lot of people's hands (not least the FSA, who have made it clear that this plus the Directive text is what IMAP is based on). I also wasn't mad keen on the reference to P&L attribution being "the real risk profile" (p48) - there is already enough flab around this term without additional mis-steers.

All in all, well worth dissecting, even if like a Roast Pigeon, you end up throwing away more than you consume!

PS - Far be it from me to suggest a document from one of the Big 4 has a whiff of Google Translate about it, but this was authored almost entirely by PwC France it would appear, perhaps explaining the odd COSO-ERM reference on p17, the rather oblique title, and of course, the whopping 60 pages...

PPS - Wife is French, so allowed the odd cheap gag around bureaucracy!