Internal Model Validation - the "desire for certainty"

Some useful snippets on the links here for anyone in the model validation space, whether if be the practical applications of Monte-Carlo simulation outside of the insurance industry, actuarial perspectives on model risks themselves such as parameter uncertainty and goodness of fit testing where "the problem is more often too many candidate distributions" as opposed to restrictions in choice.

This fantastic blog post from one of Willis's finest is about as blunt a critique of actuarial modelling activity and its potential for subsequent misuse as I have read, and I would strongly recommend it on to non-expert risk practitioners who may one day find themselves in the model validation/use test firing line. A few of the pearls of wisdom offered (focused on reinsurance industry, but relevant to all) include;

• How the human "want to believe" and "desire for certainty" can lead to models making rather than guiding decisions
• Reliance of models on "large numbers of heroic assumptions"
• "Data is always limited and flawed"
• That "models take combinations of assumptions and torture them to come to conclusions"
• The revisiting of assumptions only when the answers don't fit expectations ("euphemistically called 'calibration'", hilarious!)
• That using models for setting regulatory capital, rather than just informing decision making, has led to "extremely onerous" IMAP activity i.e. the limitations noted above are so well established that the regulators cannot ignore them at a granular level.

Then there this piece from Deloitte US on model validation, or more specifically, research into the quality of existing actuarial modelling controls, is an eye-opener for anyone working in the validation space. With RMORSA and associated capital modelling firmly on the agenda Stateside, it is interesting to watch how aggressively they approach validation, bearing in mind this work was commissioned by the Society of Actuaries, whose members may ultimately be charged with applying some of these recommendations!

This research in particular assesses current state versus best practice controls over the assumptions, inputs and outputs of actuarial models, and though the sample of respondents to the survey is relatively small (representing "30 unique companies"), the absence of suitable supporting documentation around model governance so evident in the UK's IMAP process appears to be a depressingly constant theme. This report at least includes recommendations as to how the US actuarial profession may bridge some of the gaps Deloitte identify.

In the NAIC's ORSA Manual, they ask that "ORSA Summary Report should provide a general description of the insurer’s process for model validation, including factors considered and model calibration" (p7), which I guess is what one expects to see in the EU (i.e. validation being a sub-process of the ORSA, which can be summarised in the ORSA reports). That said, the breadth of validation work performed over there will surely be driven by S&P expectations communicated in ERM Level III reviews, rather than profession-sponsored consultancy recommendations!


Finally (and slightly off track), an odd piece from Towers Watson on validating ORSAs, pitched to a room full of Internal Auditors. Would be unfair to say there aren't some salient points throughout, but given that there is "no clear requirement" to validate ORSAs (there was something on the matter in the original CEIOPS ORSA pre-consultation, but it was dropped in the public consultation and the final advice), then you would think it could be covered in less than 30+ slides!

As it happens, the TW slide pack for internal model validation appears to have been raided and had the acronym 'ORSA' jemmied into the text for much of the second half of it.

Towers Watson on US ORSA, Economic Capital and modelling trends

Towers have released a few decent bits of material of use to risk practitioners over the last couple of weeks which are worthy of comment. One on Economic Capital for Life Insurers is effectively a sales aid for their RiskAgility modelling software, but actually captures the drivers behind the UK's efforts to improve their ICA models to meet Solvency II requirements.

In particular the references to how firms ought to be making their model output 'useful' where they currently fall short (capital by business/risk/product, daily runs without running ALM models and ability to produce "what if" analysis) should be featuring highly on the agendas of both embedded use practitioners and AMSBs during 2013. Of course the sad part for any users of the software comes with the statement that RiskAgility is built "...specifically to deliver monte carlo simulation of 1 year VaR economic capital" - love to hear how the lack of multi-year is being dealt with in firm's ORSAs!

A second publication on ORSA preparedness in North America is also worth a read, even if only for us EU-based practitioners to have an opportunity to live vicariously through a country which will actually get it implemented! It is a relatively small quantum of respondents (mostly CFOs), and around half think they will be exempt on size grounds, but the perceptions which emerge are still valid, and one should be prepared to encounter this either side of the Atlantic;

• 21% see it as a compliance exercise, while 60% think it will improve ERM and capital/strategic planning
• Only 22% see their prevailing ERM frameworks tightly liked to strategic and capital planning
• Only 40% are ready to implement an ORSA in the next 6-12 months
• Concerns remain around resource requirements for educating "key personnel" and directors/C-suite - 45% and 66% respectively felt they have work to do in this area without necessarily having enough staff to do so.
• 13% of respondents didn't see their risk management departments contributing to the ORSA process (I'll get my coat then...)
• 3 year projection of capital requirements is the most common planning period envisaged

The same North American slant is then given to a financial modelling survey, which gives us another chance to peek over the fence. They found the following;

• Reasonable amount of dissatisfaction around run-times
• Around half planning to change their model governance processes in the near future

Looks like the NAIC/EIOPA covergence work should be a walk in the park then, at least on these topic...

Deloitte with more on the US-of-ORSA

Billed as a "regulatory guidepost to the future", Deloitte in the States have published their thoughts on ORSA developments, following on from recent activity in the space, most notably the NAIC's adoption of the RMORSA Act a few weeks back.

Hard to tell whether Deloitte have borrowed much from their European counterparts, who ponied up with the EIOPA-compliant equivalent document last week, but both documents ultimately point at the same end goal, namely getting the ORSA Process and ORSA Report content right.

Confidently declaring the first regulatory filing of an ORSA Report to be precisely, errr, "Sometime in 2015", the stateside plans are anchored more to ERM and, I guess by association, ratings agency implications. The document does help identify a couple elements which, with the Solvency II hat on, are easy to forget;

• IAIS ICP 16 is bringing ORSA to the table of all signatories at some future juncture (which means I may get a job back home one day!)
• Existing techniques for monitoring solvency, even in a jurisdiction of this size, are seemingly past their sell-by-date in terms of both content and turnaround time (p2) - holds true for many of the Solvency II-covered countries as well (plenty on that topic in here).

The rest of the document draws out the preparatory work which firms should be undertaking, despite the relative lack of certainty at this point in time, such as increasing real-time data availability and changes in reporting, management and governance structures. It also touches on suggested content, process implementation (more like formalisation from experience), and a checklist of operational considerations, resourcing (or even briefing/coaching) being highest priority in my mind in 2012.

Good document for you statesiders to pass round your friendly non-executive directors anyway, as an early socialising of the concept in this format goes a long way when you have tiny windows to educate them on the topic over the next 3 years - looks like you will be filing ORSA Reports before we are!

Interesting footnote is that AIG have been labelled as a potential SIFI today - ORSA may be 5 years too late to have saved the behemoth it once was, but let's hope it can help its slimmed down current-day version.

USA and ORSA - Let's do it baby!

Promising sounds from across the pond, as the NAIC make some definitive movements (detail here and here) towards the inclusion of Own Risk and Solvency Assessments as part of their regulatory reporting package - difficult to find the exact paperwork, but the summary of what was tabled by the ORSA subgroup at their August jamboree is here, while a statement to cover their adoption of the Risk Management and Own Risk and Solvency Assessment model act is available here.

I have touched on the movement of the US towards production of ORSAs in earlier blog posts, including recently on the preparedness of firms to meet ORSA reporting requirements,

From another earlier post, I can see the 15 volunteer company pilot which was mooted at the start of the year ultimately became 13 companies by the conclusion of the pilot (no word on who started, but couldn't be bothered finishing!). According to the Clearwater information, only 8 participants ORSA Reports were considered "complete", this despite the NAIC providing an ORSA Manual from which to work from.

Is this ratio of incompleteness indicative of what the 27 EU national regulators are likely to encounter in 2014, or is the lack of prescriptive guidance at Level 1 and Level 2 handy in this regard (i.e. ORSA Supervisory Reports will be "passed" regardless of quality, and regulatory arbitrage is back on the agenda).

Either way, Clearwater also note that 2015 is looking like the most likely time for imposition of ORSA reporting, so the guys will still have time to borrow from our experiences, as we can from theirs - suggested improvements  from the NAIC's sub-group to the participants (none of which would hurt anyone in the ORSA space over here!) include;

• Include a summary of “significant changes” from prior year
• Provide additional detail regarding risk managers and compensation
• Include additional stress testing, specifically for liquidity

 Looking forward to the sub-group's next report, the approach over there appears to be relatively easy to follow and well led, which I guess it can afford to be if it isn't masquerading as something other than a filing requirement... 

StoneRiver Financial Regulatory Survey - ERM and ORSA in the US

An interesting US perspective published today touching on the near term future of ERM and ORSA reporting in the States from the guys at StoneRiver (need to fill out a little form for the download).

They are of course attacking it from a "buy some reporting software" perspective so the questions are a touch loaded, but the findings from the survey (68 in the sample, majority of P&C insurers) are certainly sobering for the NAIC, namely;

• Only a quarter were confident enough to state that they had a formal process for ERM, including reporting
• 44% claim to have "in-house expertise" on ORSA, despite the requirements being in a state of flux (guess I'm not getting a green card anytime soon!)
• Only 40% were confident that their existing software will efficiently handle ORSA reporting requirements

Bearing in mind when I blogged on the draft NAIC ORSA manual in November the regulatory filing aspect was heavily emphasised, the combination of undocumented ERM processes and software fallability may be cooking up an administrative nightmare for the insurance industry over there - welcome to our world!

NAIC's Spring National Meeting - "Go ORSA, it's your birthday"

Kindly summarised by these lads, the NAIC's Spring National Meeting seems to have breathed life into the Stateside ORSA project, with the ORSA Manual (from p12) tabled in November now approved by the plenary, and a 15 company ORSA pilot group due to report at end of June. Feel free to drop me a line if you want any ideas! The legislative side of it (through an ORSA Model Act) is mooted to take a year, but from what point it is not clear - this article seems to have plenty of insight in this regard however.

Also relative was the NAIC's desire to set up an ERM training program for staff - sounds like a smart idea if ORSAs are in the offing, so should be interesting to see what they come up with

ORSA - Stateside view from NAIC's "ORSA manual"

Having already dropped a few words down on EIOPA's new ORSA consultation for us Solvency II folk, I also spotted that the guys in the States have also made a few moves with regards to their ORSA Manual (from p12 of this NAIC bundle from their meet up the other day).

What had jumped out at me earlier last week was this article quoting AIA's assistant general counsel which very much framed the US ORSA as a periodic filing requirement with which he had "privacy concerns", voiced that it “should not be compulsively adopted”, and was very much a poor cousin to the existing framework.

This of course flies in the face of the ambitions of the Solvency II ORSA which, while clearly creates a regulatory filing requirement, is intended to demonstrably drive the decision making of the undertaking.

Having had a leaf through the ORSA Manual, a few things jumped out;

• Primary goals are different to the Solvency II version (though both drink from the IAIS ERM ICP cup, the US ORSA stays on message with ERM effectiveness).
• Not compulsory, rather a requirement to be subjected to (subject to minimum premium income limit of $0.5bn as entity, or $1bn as a group)
• Can also be imposed on entities who would be considered on the "ladder of intervention" in Solvency II-speak
• Allows for cross-referencing documents, rather than summarising within an ORSA report (which gives it the look of a Pillar 3 document on Solvency II side)
• Potential option to satisfy US regulatory demands for ORSA by using any which are produced for "non-US jurisdictions" - see, Solvency II preparation is not a waste of energy for any groups with subsidiaries over there after all!
• Great comment that, wishing to use IAIS ICPs to assess the quality of non-US ORSAs, that "one of the NAIC's goals is to avoid creating duplicative regulatory requirements for internationally active insurers".
• Very concise prescriptive summaries of content - agin, they veer towards Pillar 3 requirements over here (SFCR in particular, though I haven't been through today's Pillar 3 consultations yet, so they may have changed!), but these are still very useful for people on both sides of the Atlantic.
• They go with Credit, Liquidity, Operational, Market and Underwriting risks as main "material risk categories". I'll come on to the fantastic paper on risk classification I recently attended the public rollout of later, but it has a lot of relevance to how one might approach this aspect of the proposed ORSA, including avoiding any double coverage.
• No prescribed suite of stresses, such as for the SCR standard formula - this seems to imply that "internal models" are either obligatory, or are considered to be whatever is in place currently (i.e don't need a phenomenal amount of resource to validate to the authorities). The regulators therefore are able to "provide input" into model component calibration if they don't like what is presented via ORSA.

Would love to hear from anyone over in the States to see how this is going down industry-wide, as I can't quite gauge how much of an additional imposition this is to business as usual!

Solvency II Internal Model Validation slides - from the US!

Of use to anyone working on internal model validation, I clocked this presentation from one of Ernst and Young's finest across the pond. Some useful context within it, particularly showing where the E&Y guys have come across gaps in validation policies, methodologies, documentation.

This is all in the context of the States' own preparations for ORSA under the guise of the NAIC (indeed I touched on it the other day), but is of course very handy on our side of the Atlantic!

NAIC, ORSA, ERM and SRQ - Acronym day!

Saw this nice piece from Towers Watson on ERM developments Stateside (I had blogged on this from an ORSA perspective a couple of weeks ago). The pressure from the AM Best ratings agency seems to be pretty intense to this regard, and indeed Towers Watson's article here highlights what AM Best consider "leading" and "lagging" practices.

"Leading" practices have all the hallmarks of the IAIS/Solvency II world that we all know and love ("tolerance", "appetite", "risk-adjusted return", "modelling used in strategic decision making" etc etc). What is more enlightening is the section on "lagging" practices, as it seems pretty bold to say that by not adhering to the AM Best standards you are a laggard.

However, the stats are pretty sobering (and mouthwatering if you are consulting over there!). 90% of respondents' risk tolerance definitions were "too broad", 12% had no ERM Committee/CRO and 72% do not use internal models.

Towers (who have issued guidance on the 2010 iteration) go as far as to say that the new ERM section on Best's SRQ (Supplementary Ratings Questionnaire) will '...become more influential on ratings outcomes' and that this evolution will compel insurers to 'stay ahead of their peers and Best's evolving ERM criteria'. How the SRQ obligations marry into the ORSA guidance due out soon is another thing, but fascinating times ahead for the North Americans...

NAIC - ORSA and ERM developments in North America

Spotted this over on the Casact website, and though it was worth sharing - emerging ORSA approach over with our North American chums at the NAIC, which seems to be spooking some insurers who are quaking at the thought of a 2012 ORSA imposition and lobbying for more time (same on this side of the Atlantic fellas!).

Seems to be a bit of draft ORSA guidance on the go (along the same lines as our Level 3 document in the EU), and the one stand out piece for me was the projection of capital requirements between 2-5 years out - seems instinctively light for insurers to project no further than 5 years, but maybe the long-term guarantee piece isn't such an issue Stateside.

There is also an enlightening piece from AM Best on the same document regarding their Supplementing Rating Questionnaire, how best to prepare for it and respond to it, and some benchmarking stats on positive responses to the questionnaire content. Sadly I don't get to dabble too much in this area, so if any of the Stateside readers have any experiences to share on ORSA progress or ERM SRQ completion I would be interested to hear.

Insurance Day Summit 2011 - Bermuda shorts

Exceptional pun in the headline aside (?), there was a few tasty pieces from the Insurance Day summit in chilly Bermuda at the end of last week. Not sure if these are free or not, but I didn't struggle this morning;

Bermuda Monetary Authority
- their man Jeremy Cox spoke, emphasising how mutually beneficial the Bermuda/EU market is, and why it justifies increased headcount spend to ensure equivalence. He also spoke of their 3 year roadmap to equivalence, as well as 6 areas where they have expanded their output to achieve equivalence.

Mike McGavick speech
- CEO of XL Capital chipped in with a more industrial quality speech, albeit making the salient point that, should Solvency II inadvertently lead to mergers and larger insurers, it would not be a systematically great thing.

He comments (ironically, bearing in mind my last two posts about soft launching) that because the EU like their version of the rulebook, then "everyone else should follow suit".

He followed that with "...why is it that we just have to swallow your [the EU's] proposal" - again, this seems a little misguided, bearing in mind the IAIS have xeroxed most of the Solvency II text and obligations, so this is not far off being international best-in-class, as opposed to being EU-centric.

Certainly didn't stay on the fence, and should be applauded for that at least.

NAIC CEO speech

Pretty blunt speech from the head of the National Association of Insurance Commisioner in the US - that, while she likes soome parts of Solvency II ("ORSA and some of those Pillar II things"), the Pillar I stuff will not wash over there. In particular, quite scathing about regulatory capital being used to incentivise good risk management.