|
|
Do You Need an Independent, Third Party ALM Model Verification?
By William J. McGuire, Ph.D., President and CEO, MPS
ALM models are widely used by banks, thrifts, and credit unions. The outputs from these models guide financial managers in making decisions on topics ranging from defining liquidity requirements to examining how balance sheet strategies determine immediate and long-term interest rate risk (IRR) exposures. The scope of ALM model usage leads to the concern that model error—basing crucial decisions on incorrect model forecasts—will adversely affect performance or create undue IRR exposure.
To properly control model risk, independent, third-party assessments are often needed. This article examines when such a solution is required to meet the now-stricter regulatory mandates and create confidence in an ALM model as a balance sheet management and IRR analysis tool. The discussions presented empower financial managers to make the right decision regarding their institution’s specific need for an independent, third-party, model risk assessment.
The Elements of a Model Risk Assessment Solution
The potential for model error—and the necessity to manage that risk—create the need for a formal assessment solution in most financial institutions. Regulatory guidance is available in OCC Bulletin 2000-16 (the current guiding directive on ALM model verification for national banks and a leading industry resource in the area) and from the FDIC (December 7, 2005, Supervisory Insights article on model governance). Other agencies have also issued advice—for the most part derivative from the OCC bulletin. There are also strong business reasons to assess and control model risk. Most important is the need to avoid the performance surprises that can arise from inaccurate ALM model forecasts. Another key rationale is that a fully functioning and highly accurate ALM model increases the precision of financial decision-making, leading to higher levels of performance and value.
Acceptable assessments of model risk prove that an ALM model: (a) accurately captures the behaviors described in the financial contracts underlying the balance sheet’s asset and liability categories, (b) precisely forecasts performance and present values at the category level, and (c) is surrounded by an adequate model-control environment and governance solution. As background, consider each of these elements in more detail.
Model verification establishes the theoretic capability of the model to forecast accurately. Input data exactness and footing at key control points in the model, appropriate model set-up attributes (which define how the model acts on incoming data), contractual inputs (e.g., caps, floors, pricing indexes and spreads, puts/calls that can be defined from record level data), behavior assumptions (e.g., loan prepayments and core deposit repricing and runoff), and reporting accuracy are all affirmed in this step.
Model validation proves the actual ability of the model to forecast accurately. This is done by reviewing model outputs in specialized diagnostic systems that demonstrate the correctness of category level model forecasts across interest rate scenarios relative to specific underlying contract specifications. First-month yield and cost forecast values versus prior-month margin data are also typically compared. As possible, the correlation of model IRR predictions with trends in historic performance and value are also reviewed to provide empirical analog-based validation.
Model control environment and model governance review establishes the quality of data input and user controls, and assesses the adequacy of the Asset/Liability Management Committee (ALCO) solution surrounding the model and its applications. Quality control procedures, user checklists, policies, procedures, and other elements are reviewed.
When is an Independent, Third Party Solution Required?
The need for an independent, third-party model risk assessment is determined by how each institution’s situation aligns with the underlying requirements of the activity. The first mandate for a model risk assessment is that the party performing the assessment has the high levels of expertise and broad industry experience needed to conduct the comprehensive verification, validation, and process reviews described above. Requisite levels of model expertise are in most cases present internally (reference, however, the independence discussion below), but broad industry experience is often lacking.
The next mandate is that the party performing the model risk assessment must be independent from the function of running the model or interacting (directly or indirectly) with its outputs. This eliminates staff involved in using the model to produce forecasts, who are normally the only internal personnel with high levels of modeling expertise (and perhaps industry experience). It also excludes ALM model vendors who are reviewing their own models, bond brokers, wholesale funding providers, and many ALM consultants (most notably those providing advice based on model outputs).
A final mandate for model risk assessments is that the party providing the service be outside the realm of institutional politics or other entangling conditions. This usually rules out almost all other internal staff not otherwise excluded, except perhaps for those in the institution’s internal audit group. The audit function, however, rarely has the expertise or industry experience needed to complete all facets of a model risk assessment.
In light of the mandates discussed above, a simple rule emerges: An independent, third-party, ALM model-risk assessment provider is required when internal resources cannot meet the criteria discussed above. Note: An external assessment may also be a desired solution when the internal audit group or management wants the assessment to be conducted by an external third party to obtain a view from outside the institution.
How Can I Control the Costs of Model Risk Assessments?
Acceptable quality, independent, third-party model risk assessments are not inexpensive. This is because verifying and validating the model, and examining its control environment and governance processes, are labor intensive. There are no shortcuts here, such as simple checklists or interview surveys, because accuracy and precision in an ALM model reside in the details. But budgets being what they are, your checkbook likely has its limits. In a world where regulatory guidance is drifting toward mandating annual independent model risk assessments, are there any actions that can be taken to reduce the cost of a model-risk compliance program? The answer is yes, but only sometimes.
To set the background for a cost reduction plan, inventory all ALM model applications (e.g., liquidity monitoring, IRR analysis, business planning, and budgeting). Fewer model applications and fewer mission-critical uses of model forecasts, create less of a need for frequent model risk assessment. Smaller institutions and those with high CAMELS ratings can also typically engage in less intensive model-risk assessment programs. This is because their model risk is less of a direct threat to the deposit insurance funds.
A way to reduce out-of-pocket costs is to engage internal audit as an integral part of the model-risk assessment solution whenever possible. Comparative advantages of internal audit staff are often found in creating a model application inventory, data audits, ALCO policies and procedures reviews, control environment design, and ongoing monitoring.
The need for an annual model risk assessment is controversial. Banking sector regulators (and the NCUA tends to follow the FDIC) have expressed a strong preference for annual independent assessments. Where the balance sheet has changed significantly (e.g., with an acquisition), after major staff turnover, or when economic or interest rate conditions are fast changing, annual model risk assessments can be justified. But, in the opinion of this author, annual assessments should not generally be the case. A much more cost-effective way to create ALM model precision would be to conduct a model risk assessment every two or three years, devoting the off year(s) to statistically quantifying institution-specific loan prepayment and core deposit behavior and value inputs.
Even if an annual model risk assessment is a regulatory requirement, there is a way to reduce cost over a two or three year cycle. In the first year, conduct a comprehensive model verification, validation, and model control environment/governance review. In the off year(s), concentrate on verifying and validating the model at a purely technical level. This is a smaller deliverable, and as such, it can be priced lower by the provider.
Be sure to include a clear description of any chosen, non-annual model-risk assessment program in the ALCO policy (with a clear specification of conditions triggering more frequent comprehensive model risk assessments) and obtain board approval.
Where is the Model Risk in Most ALM Models?
The history of formal ALM model risk assessment mandates goes back almost 10 years, and informal programs existed prior to that. As a participant in model risk assessments since their beginning, the author has watched significant changes unfold in both ALM models and their applications. Across time, some progress has been made, but new challenges continually emerge.
Ten years ago, the key problem in ALM models was data. Links to underlying systems were primitive, and many manual overrides or entries were required. But most model applications were simpler in those days, before equity-at-risk IRR analysis was mandated. The regulatory climate was also more benign, as model risk was of limited magnitude.
Today, data intake into an ALM model is a mechanical chore rather than a daunting challenge. But balance sheets and model applications have become vastly more complex. This has shifted the typical areas of model weakness. Now model risk is found in the failure to capture option-related characteristics (e.g., calls and puts embedded in investment and wholesale funding, loan prepayment across many categories, and CD “bump up” and early withdrawals) and inadequate core deposit behavior inputs (e.g., supply/liquidity paths, repricing magnitude and lags, and term/present value related behaviors). Equity-at-risk IRR analysis specifications are now another weak point for many model applications.
Along with more complex balance sheets, regulatory concerns have mounted as the potential magnitude of model risk as a threat to the deposit insurance funds has increased. This has brought a stronger focus on model risk in general and specific recommendations for additional exposure testing. An example of the latter is the now common suggestion to use statistically derived rate ramps to assess basis risk (i.e., through the unique speeds at which driver rates change or via such ramps where the yield curve shape changes). Too few institutions do such testing, opening themselves up for performance surprises.
Changes in the broader regulatory environment (e.g., Sarbanes-Oxley) have also been factors in altering the regulatory mandates relating to ALM models. This is especially seen in the scrutiny given the model control environment and ALCO governance. What were acceptable data and user controls, policies, procedures, and reporting only a few years ago, no longer pass muster. But many institutions have been slow to adjust, and many are operating with inadequate model controls and deficient governance solutions.
The good news is that progress is being made toward more accurate ALM models. Improvements will likely occur faster where independent, third-party model-risk assessment providers are involved. This stems from an advantage of external reviews, which bring wide industry experience to each assessment. Industry best practices are more effectively spread, and unique solutions generalized, as the provider’s experience base expands and new insights are shared with institutions. Depending on needs, this may be another reason to consider an independent, third-party provider when defining a model-risk assessment solution.
Printable Version
E-mail this article
 |
|
Newsletter content is for our readers' informational purposes only.
Please refer to our Terms of Use for details. |
|