Keep Up With the Standards: On ASOP 56, Modeling
By Mary Pat Campbell
The Modeling Platform, April 2021
On Oct. 1, 2020, Actuarial Standard of Practice (ASOP) No. 56, Modeling, went into effect for actuaries practicing in the United States. As is usual with standards-setting, this new ASOP was years in the making: in 2010, the Actuarial Standards Board requested the Life Committee work on an ASOP focused on modeling. While the Life Committee was originally asked to develop the standard, ASOP 56 applies to all actuaries (excepting some specific pension-related work).
Before this new modeling ASOP, the only explicitly model-related ASOP was ASOP No. 38, Using Models Outside the Actuary’s Area of Expertise and that was focused on property-casualty actuarial work, with the concept that some P&C actuaries needed to use the results from catastrophe models, which is outside standard actuarial expertise. The scope of ASOP 38 explicitly applies to actuaries
“who use models that incorporate specialized knowledge outside of the actuary’s own area of expertise when performing professional services in connection with property and casualty insurance coverages (including risk financing systems, such as self-insurance and securitization products, that provide similar coverages).”[1]
In contrast to the scope of ASOP 56:
“This standard applies to actuaries in any practice area when performing actuarial services with respect to designing, developing, selecting, modifying, or using all types of models. For example, an actuary using a model developed by others in which the actuary is responsible for the model output is subject to this standard.”[2]
That scope is comprehensive. One can argue that all the numerical work actuaries do, no matter which field, involves modeling of some sort, even if only simple calculations in a spreadsheet.
Areas Covered in ASOP 56
By their very nature, ASOPs are high-level descriptions of what is expected of actuarial work. They lay down expectations in terms of the depth, breadth, and quality of what we deliver to intended users of our work.
Let us focus specifically on Section 3 of ASOP 56: Analysis of Issues and Recommended Practices.
To give a taste for what is involved, here is subsection 3.1:
3.1 MODEL MEETING THE INTENDED PURPOSE
The actuary should understand the model’s intended purpose.[3]
To be sure, there are multiple sub-subsections underneath this subsection header:
3.1.1 Designing, developing, or modifying the model
3.1.2 Selecting, reviewing, or evaluating the model
3.1.3 Using the model
3.1.4 Model structure
3.1.5 Data
3.1.6 Assumptions used as input
Each of these sub-subsections are with respect to the intended purpose of the model. For example, underneath 3.1.4 Model Structure, we have the following text, which I have elided somewhat:
“The actuary should assess whether the structure of the model (including judgments reflected in the model) is appropriate for the intended purpose. The actuary should consider the following, as applicable, for a particular model:
a. which provisions and risks specific to a business segment, contract, or plan, if any, or interactions more broadly, are material and appropriate to reflect in the model;
b. whether the form of the model is appropriate;
c. whether the use of the model dictates a particular level of detail;
d. whether there is a material risk of the model overfitting the data; and
e. whether the model appropriately represents options, if any, that could be reasonably expected to have a material effect on the output of the model.”[4]
Looking just at that sub-subsection of ASOP 56 reminds me of all the modeling I’ve seen over the years that failed at least one of the five elements mentioned above.
Use Appropriate Models
In particular, I have been involved with the Academy Interest Rate Generator for many years through Academy work groups and then an SOA Project Oversight Group (POG). This model was developed for a particular purpose, and then parameterization was changed because of regulator (not necessarily actuarial) input. The structure assumes specific correlations between specific economic items (including assumption of independence of other items). It is a stochastic model calibrated to work with real-world probability and real-world interest rate yield curve dynamics.
In addition to having parameters changed by regulators, we also had to deal with the external circumstances of extremely low interest rates which were not reflected in the original model structure. The structure does not allow for nominal negative interest rates, which could potentially be a problem in the future.
Because this very specific model is freely available (see here: https://www.soa.org/resources/tables-calcs-tools/research-scenario/, Economic Scenario Generators), we have often received questions asking about adapting this model for use outside the scope that the model was originally developed for.
Mind you, some models can be adapted for use from one situation to another. It requires understanding the structure of the original model and determining what, if any, adjustments need to be made. However, sometimes you can’t get from there to here.
Now, the next time somebody asks if options can be priced with this real-world-probability interest rate model, I can point to ASOP 56, and in particular, subsection 3.1.
The Need for Modeling Standards
The argument can be made that we did not need yet another ASOP. Don’t all actuaries know that they should pick appropriately-structured models? Don’t we all know that we should test our models in multiple ways, to make sure they actually calculate what we intend?
While this is true, modeling is becoming more complex, and involving more non-actuarial professionals, than ever before. Modeling is getting increased scrutiny in financial reporting, with IFRS 17 and the long-duration contracts targeted improvements from FASB. Predictive modeling is being used in ever-more applications within insurance, and that can affect actuarial modeling even if the actuaries are not the ones directly involved in underwriting.
If nothing else, having a checklist to go through while working on modeling can help you make sure you don’t miss anything. Hey, ASB, make some handy-dandy sticky note checklists we can stick on our monitors to ask us:
3.1 Does our model meet the intended purpose?
3.2 Do we understand the model, especially any weaknesses and limitations?
3.3 Are we relying on data or other information supplied by others?
3.4 Are we relying on models developed by others?
3.5 Are we relying on experts in the development of the model?
3.6 Have we evaluated and mitigated model risk?
3.7 Have we appropriately documented the model?
Yes, the above list comes from the subsections under Section 3, Analysis of Issues and Recommended Practices. And maybe the ASB can also put in the subpoints for 3.1 and 3.6 explicitly, such as “Have I validated model output?”
One of the biggest reasons we promulgate professional standards for all U.S. actuaries is to maintain the integrity and reputation of the actuarial profession. As Tom Wildsmith wrote, “Without meaningful, binding standards of practice, the profession would not have been able to earn the public’s trust.”[5]
There is nothing precluding statisticians, “data scientists,” and other non-actuaries from following ASOP 56. There is nothing precluding non-U.S. actuaries from following this standard. The list of considerations is good for any serious modeler to follow, whether it’s asset modeling, predictive modeling, machine learning modeling, or traditional life insurance liability modeling.
However, should various non-U.S. actuaries fail in following ASOP 56, there may be personal fall-out from such a failure (or more serious fall-out, such as when a spreadsheet formula error understated risk exposure by 50 percent, leading to a loss of billions of dollars[6]), but no formal procedure for investigating and potentially removing their credentials.
If an actuary fails to follow the ASOP, there could be a hearing by the Actuarial Board for Counseling and Discipline (ABCD), which can lead to expulsion from actuarial organizations and a loss of credentials at the most serious end. That is an extreme example, and the outcome of most consultations with the ABCD is counseling, and guidance on how to better follow the ASOPs.
Perfection is not expected in following the ASOPs, and perfection is probably unattainable anyway. However, in following the various aspects of ASOP 56, we will surely make our modeling product that much better.
Go and read ASOP 56 now (and don’t forget to take the CE credit)! Go forth and spread the standard for others to follow! Let us improve modeling practice for everybody!
Statements of fact and opinions expressed herein are those of the individual authors and are not necessarily those of the Society of Actuaries, the editors, or the respective authors’ employers.
Mary Pat Campbell, FSA, MAAA, is vice president, Insurance Research, for Conning Research & Consulting, Inc. She can be contacted at marypat.campbell@gmail.com.