Scores for Valuation Reports: Appraisal Score & BPO Score White Paper APRIL 2012 White Paper
Table of Contents Overview... 3 Generally Accepted Appraisal Rules... 3 Development of Rules... 3 Coding of Rules... 6 Quality Assessment of the Rules... 7 Identification of Incorrectly Firing Rules... 7 Documentation... 8 The Appraisal Score... 8 Score Development... 9 Distribution of Scores... 11 Version Control... 14 Testing and Audits... 15 BPO Score... 15 BPO Form... 16 Score Development... 17 Score Reporting... 17 Appendix - Example Test... 19 Appraisal Score... 19 We know the score. www.collateraldna.com 2
Overview In 2004 FNC released the Generally Accepted Appraisal Rules (GAAR) to provide lenders with the ability to enhance the review process by automating the preliminary appraisal review. These rules have gone through a series of enhancements over time. In November 2005 they were completely rewritten in version 4.0 to address the new Fannie Mae and Freddie Mac appraisal forms. In 2010 the external data validation rules and the FHA compliance rules were added (version 6.0) and in 2011 the GSE UAD compliance rules were introduced (version 6.1). The rules now consist of; a) compliance rules to check for USPAP, GSE and other regulatory requirements, b) risk rules to identify areas of concern in the appraisal related to the property, inconsistencies, and potential fraud issues, c) external data validation rules to check the data in the appraisal against data from FNC's National Collateral Database, d) FHA and VA compliance rules and e) GSE UAD compliance rules to validate that the appraisal is compliant with the new GSE requirements for data delivery. Originally in 2007 and then more recently in 2011 FNC also developed rules for the review of BPO forms. Unlike appraisals, there are no standards for the BPO forms and therefore each form has a unique set of data. In 2011, FNC developed an FNC BPO data standard which attempted to provide a common set of data that could generally be found on most forms. Further, FNC created a recommended BPO form and wrote rules around that form. Along with the development of the rules for the review of the appraisal, FNC also developed the Appraisal Score based on the GAAR rules to assist lenders in classifying appraisals as to their risk of having valuation problems. The development of the score and its general performance is described in the following sections. Recently, in 2011, an additional score has also been developed for broker price opinions (BPO). A description of that score is available in the score section of this paper. Generally Accepted Appraisal Rules The Appraisal Score is based on FNC's Generally Accepted Appraisal Rules (GAAR). These are detailed rules to evaluate the appraisal. They consist of an extensive set of rules for each form type. The rules were developed by expert appraisers at FNC based on industry "best of breed" practices. Development of Rules To create the review rules for the appraisals, the rules were initially categorized into two groups; compliance and risk. Expert appraisers created all of the rules. To develop the We know the score. www.collateraldna.com 3
compliance rules, the USPAP regulations, the Freddie and Fannie guidelines, and the rules of various regulatory agencies were consulted and translated into specific rules to ensure that appraisals passing the compliance rules would be in conformance with the guidelines and regulatory requirements. The risk rules are based on best industry practices for appraisal review along with the extensive experience of the appraisers developing the rules. The intent is to develop a comprehensive set of rules that will thoroughly identify risk issues within appraisals that may reflect on the valuation opinion provided by the appraisal. After the rules had been in production for several years, a concern developed that fraudulent data could be manufactured to make the appraisal appear better than it actually was. To address this concern a new set of rules was added to check all of the data on the appraisal for the subject and comparables with the National Collateral Database to validate the data. Further, to address the increased demand for BPOs following the housing market collapse by servicers having to deal with delinquencies, FNC developed a series of rules in support of BPO forms. Originally these were limited to completeness for a variety of BPO forms but in 2011 and 2012 were expanded to address additional issues and are now designed for the FNC BPO form. Details of the development of the GAAR rules can be found in the FNC white paper on GAAR. Highlights of the development are provided below. Best Practices - January, 2004 First set of standardized rules Compliance issues only Only one form supported GAARR (Generally Accepted Appraisal Review Rules) - April, 2004 Name Change Expand number of compliance rules GAAR (Generally Accepted Appraisal Rules) Version 1.0 - July, 2004 Name Change Add two additional forms We know the score. www.collateraldna.com 4
GAAR Version 2.1 - January, 2005 Correct bugs in rules Minor changes to rules Add additional rules for Condo form GAAR Version 3.0 - March, 2005 Added Risk Rules for URAR Made small edits to existing rules GAAR Version 3.1 - June, 2005 Added Risk Rules for 2055 Additional Risk Rules for URAR Minor adjustments to other rules GAAR Version 4.0 - September, 2005 Rewrote all rules for new Fannie Mae and Freddie Mac 2005 forms Added two additional forms Added Risk Rules for all forms Significantly restructured many rules Substantially expanded the number of rules GAAR Version 4.1 - February, 2006 Added Compliance rules for Small Residential Income Property Appraisal Report Minor modifications to rules GAAR Version 4.2 - July, 2006 Added Risk Rules for Small Residential Income Property Appraisal Report Enhanced a number of rules Minor modifications to improve performance of many rules We know the score. www.collateraldna.com 5
BPO Rules Version 0.1 November, 2007 Created initial rule set of completeness rules Based on FNC prototype BPO form GAAR Version 5.0 - January, 2008 Added FHA Compliance Rules for URAR Minor edits to improve rule performance GAAR Version 6.0 - January, 2009 Added external data validation rules to all forms both for compliance and for risk Restructured Risk and Compliance categories resulting in renaming some rules Additional edits to rules were completed to improve performance GAAR Market Conditions Report Version 1.0 - March, 2009 Created new rule set for the newly required Market Conditions Addendum to the Appraisal Report (1004MC/71) GAAR Version 6.1 UAD - July, 2011 Extensive rework of all existing rules to handle new UAD form data Added compliance rules to ensure UAD compliance BPO Rules Version 1.0 September, 2011 Completeness and data validation rules Based on FNC BPO form Once created, the rules are updated based on reviews by clients and based on their performance with production appraisals. To detect inappropriate firings of the rules detailed audits are conducted each quarter. Coding of Rules Each rule is coded so that the XML data that is delivered from the data extraction source can be run by an independent program. The rules are designed so that the rules will run exactly the same whether the data comes from OCR, PDF extraction or from AIReady files. All code is We know the score. www.collateraldna.com 6
managed through Visual Source Safe software and the code development process is managed through the use of RAID, an FNC development tracking system. Quality Assessment of the Rules All rules are reviewed and tested in development, QA, and UAT environments prior to moving to production. Once the rules are coded, their performance is evaluated through a series of test scripts on a large number of XML data files that represent extracted appraisal data with a number of known issues. Each rule for each form is evaluated to ensure that it is performing correctly. The purpose of the QA testing is to ensure that the rule, as coded, accurately performs as designed. Problems that are identified are reported back to the development staff through the RAID tracking system. The test scripts are built up to detect known issues and include items that result from feedback from the field about problems with the manner in which the rules have performed in the past. Over time the test scripts are built up to include a wide variety of issues to ensure the robust performance of the rules. Production appraisal firings are routinely reviewed to identify rules with false positive firings so that the rules can be modified to correct for the many variations that appraisers use for reporting information. Identification of Incorrectly Firing Rules Once all of the rules are performing as designed the rules are run against a very large data set of appraisals drawn from production. These include appraisals that have been extracted by pdf, ocr or which have been delivered as AIReady files. The firing rates for each of the rules are tabulated and any rules are examined which appear to be firing too frequently. Each rule is compared with the actual appraisals that caused it to fire to validate whether or not the rule should have fired. If it is determined that the implementation of the rule was causing it to fire inappropriately, then the rule is modified to take into account why the rule was firing incorrectly. This change is added to the RAID tracking system to be coded and to go through the QA process again. There are two other ways in which rules are identified for modification. First, a number of appraisals are randomly selected from production and are compared to the rules to ensure that none fired inappropriately and that there were no issues on the appraisal that should have caused a rule to fire when none fired. It is also possible that during this process an issue might be identified that is a potential problem on the appraisal for which no rule exists. If any of these issues are identified then rules are added or modified as appropriate and sent back through the development process. Second, problems with the rules are sometimes identified by reviewers We know the score. www.collateraldna.com 7
in the production environment and reported back to FNC. These problems are reviewed and if the problem is verified then the rule is modified and sent back through the development process. The review of the rules is an ongoing process that is repeated routinely. Documentation FNC has developed extensive documentation describing the rules, the motivation for each rule, how the coding interprets the rule and the performance of the rules. The documentation also describes how interpretations of appraiser comments are handled and provides a glossary of the terms used in the rules. This documentation is updated as changes are made to the rules and all changes are tracked, described and dated. Although manuals of the rules exist, increasingly the documentation is access via a controlled access site on the internet. A user id and password provides the user access to all the rules and the ability to do online searches for key words and rule names and numbers. Rule descriptions are provided in the Appraisal Score Report and in the CMS. These descriptions are written for the appraisal reviewer and are somewhat terse and to the point. However, for the compliance rules, they are often returned directly to the appraiser and in those circumstances the shorter rule description would likely be confusing to the appraiser. Therefore, for the compliance rules, an alternative description of the rule has been written describing what the appraiser needs to do to correct the potential issue. Documentation showing both sets of rule descriptions for each rule are also available on the website. The Appraisal Score The Appraisal Score is based on all of the compliance and risk rules in GAAR. Currently, the UAD rules and the FHA/VA rules are not included in the Appraisal Score. The purpose of the Appraisal Score is to provide guidance to a reviewer as to the potential uncertainty that exists around the value conclusion. In this sense it is similar to a confidence score on the appraiser's estimate of value. Lenders often use the Appraisal Score in conjunction with specific GAAR findings to categorize the appraisals for different levels of review. We know the score. www.collateraldna.com 8
Score Development The Appraisal Score was originally developed in 2004. Then, prior to any lender being able to use the score in production, it had to be recreated when the new GSE forms were introduced in 2005 the Appraisal Score was recreated to work with the new forms and the new GAAR rules developed for those forms. This Appraisal Score was in production until 2010 with periodic modifications to reflect rule changes and updates. In 2009 with the introduction of the external data rules a third version of the score was developed and with the introduction of GAAR 6.1 all clients were moved to the new score. The development of a score for appraisals to aide in automating the process for review has two fundamental phases. When new forms are introduced as occurred with the FNMA forms in 2005, and with the new UAD version of the forms currently being introduced, there is no history of how these forms have been used in the past. Therefore, there is no data to identify how appraisers will use them in the future nor is there any data associated with appraisals that have resulted in bad valuations or losses to the lender. As a result, statistical approaches are not appropriate since there is no data to work with to estimate appropriate weights for the scores. Therefore, the approach that is used is to have experts assign weights for the rules that reflect their expert opinion as to the seriousness of the rule. These weights are then used to create scores on the appraisals. The scores are created by adding the weights associated with the rules that fire and then transforming them through the following equation. This results in scores that range from 0 to 1000 with high scores being generated by few rule firings and low scores by those appraisals with many rules that fire. Equation 1 Score Formula Score = 1+ e K n ω jγ j j = 1 Γ Where K and Γ are constants, the ω j are the weights and the γ j are either dummy variables that take on the value 1 if the rule fires and 0 otherwise or the output of sub-score functions identical to the one shown above. Once production develops sufficient examples of appraisals that have resulted in bad valuations or losses, then statistical means can be used to re-estimate the weights that are applied to the rules to better differentiate between the appraisals with value issues and the remaining production volume. To estimate these weights the appraisals are separated into two groups, those with bad valuations and those with good valuations. We know the score. www.collateraldna.com 9
Then, the rules that fire for each appraisal are used as the explanatory variables and the two groups are used to create the binary dependent variable. Equation 1 is a completely flexible form function which contains common regression functions such as linear and non-linear functions, logit, probit, exponential, log, etc. all as special cases of this function. The optimization problem for developing the score can be summarized as follows: Let the number of files be designated as N with the number of bad valuations equal to B. γ is a vector taking a value of 1 when the observation has a bad value and 0 otherwise. The total number of rules can be indicated as R and the matrix r has N rows and R columns with each component r ij taking the value of 1 if the jth rule fires on the ith observation and zero otherwise. Let K and Γ be constants. The optimization problem is then to maximize Ψ where: Ψ = N δ i IND( S i < 500) (1 δ i ) IND( S i= + B N B i= 1 1 N i 500) and, Equation 2 - Optimization Function S i K = R j 1 1 + e ω jγ ji Γ There were two major changes with the transition to the current score. First, the external data rules were included as inputs to the score. Second, there was increased interest by clients in automatically returning any appraisal back to the appraiser if it had any compliance issues. As a result, the current Appraisal Score was adjusted to reflect the change in handling of the compliance rules and to include external data validation rules in addition to the risk rules. To create the new version of the score it was important that consistency be maintained between the former and the new score since it was widely used by lenders in production. To achieve the consistency tens of thousands of appraisals from production were used and weights were estimated that produced similar individual appraisal scores and the same distribution of scores for both sets of scores on the large sample. However, following the completion of the new score collapse of the residential housing market let to a substantial decrease in the Appraisal Scores. Issues such as increased REO, increased distances to We know the score. www.collateraldna.com 10
comparables, older comparables, less similar comparables, and price declines all led to more problematic adjustments and many more rule firings. All these problems were amplified by the new external data rules which incorporated many rules monitoring market conditions. Thus, as a result of the housing market decline, the scores in general have significantly decreased. As the market recovers, it is anticipated that the scores will return to their former values. The BPO score is developed in exactly the same manner by using inputs from the BPO rules. A description of the BPO score is in a subsequent section. Distribution of Scores With the development of any new version of the score, a large number of production appraisals are scored and the distribution of the resultant scores computed. Samples are taken from both tails of the distribution and analyzed to see if the scores are accurately reflecting the quality of the appraisals. Further, the distributions of the scores are provided to clients to assist them in defining rules for automation of the appraisal review process. Due to the nature of the GAAR rules, there are several general influences on the Appraisal Score: Appraisal Scores tend to be lower in rural areas and higher in suburban areas Appraisal Scores tend to be lower in declining markets and higher in stable or improving markets Appraisal Scores tend to be higher in areas where the properties are homogeneous and lower where properties are dissimilar Appraisal Scores tend to be lower for higher valued properties and higher for moderately priced homes These large scale comments on the score are largely driven by the quality of the comparable sales that are used by the appraiser and only reflect a general trend. Specific properties can have any level score despite their location or market. The change in the overall scores due to the deterioration of the market can be seen from the following distributions shown in Figures 1 and 2. We know the score. www.collateraldna.com 11
Figure 1 - Score Distributions on Current and Pre-Market Collapse Appraisals Figure 2 - Cumulative Score Distributions on Current and Pre-Market Collapse Appraisals We know the score. www.collateraldna.com 12
To see how the value of the property impacts the score, the following graphs (Figures 3 and 4) show the distributions of the scores for all properties scored over the period January 2011 through May 2011 which had a value of more than $500,000. The vast majority of the sample were properties between $500,000 and $1 million and the sample sizes dropped significantly as the values increased. The graphs below show how the distribution is shifted to lower values as the less expensive homes are removed from the distribution. Figure 3 - Score Distribution by Value Tiers We know the score. www.collateraldna.com 13
Figure 4 - Cumulative Score Distribution by Value Tiers Similar graphs can be generated showing the impacts of rural vs suburban, and homogeneous vs heterogeneous neighborhoods. Those graphs, while generally showing the same types of shifts would typically not be as dramatic as the above graphs based on property values. As the values move into the highest valued properties it becomes more and more difficult to find appropriate comparables. That is even more true in the current market where the jumbo mortgage market has been very limited. Version Control Each rule that is in production has its own version number so that any report about any specific appraisal can be documented and reproduced as to the details of the specific rules that fired. Archives are maintained of past rules so that the specific code of the rule can be reviewed in the future if the need arises. Major rule changes will be numbered as major number changes and limited changes will be tracked as minor number changes. We know the score. www.collateraldna.com 14
Testing and Audits Each quarter FNC conducts a detailed audit of the rules and the score. A randomly chosen sample of appraisals is selected and run through the rules and are scored. Then each appraisal is reviewed in detail to determine whether the rules fired correctly and whether or not the score seems appropriate as an indicator of further review. Each rule that fired is evaluated to determine the appropriate category: Did the rule fire correctly and did it identify an appropriate issue? Did the rule fire as written but did not correctly identify a problem? (An example would be if the rule was looking for the word mold but fired when the appraiser wrote molding) Did the rule fire incorrectly? In other words was the rule not properly coded. Did the rule not fire when it should have fired? In addition, each appraisal is examined to attempt to identify any area of concern on the appraisal for which there does not exist a rule to identify the problem. All of the findings are described in detail in a report that is made available to each client using the Appraisal Score. FNC also monitors firings of rules and scores across all clients to identify unexpected changes in either the score distribution or the rate of firing of any of the rules. Reports are generated monthly describing the rule firings and score distributions. Many clients also conduct audits and tests of the score and rule performance. The nature of these tests are varied ranging from an analysis of the the degree to which the score can identify problem appraisals to an evaluation to what extent the score can predict post -closing loss on a loan. Most of these tests are proprietary and cannot be shared. As example of a test from several years ago that a lender agreed to share is provided in the appendix. BPO Score Loan servicing and capital markets have always used Broker Price Opinions (BPO) as their primary source of property valuation information. With the collapse of the real estate market the demand for BPOs began to increase significantly and correspondingly the need for some means to automatically review BPO reports and assist reviewers with the review process. The challenge in creating a BPO score is that there are no standards for the form to be used, or the data incorporated on the form or for how the data is recorded in the form. As a result, rules cannot be designed which will work on all forms. Therefore, to enable a BPO Score to be We know the score. www.collateraldna.com 15
created, FNC created a BPO form described in the next section. The BPO Score was developed in an identical fashion to the manner described above for the Appraisal Score but based on the data and rules defined by the FNC BPO Form. Other forms can be scored as long as they contain the same data fields which are in the FNC BPO form. BPO Form The FNC BPO Form is shown in figure 5 below. In order to develop the form a large number of BPO forms currently used in production were examined and common data elements across all forms was used as the base information. Then servicers were consulted about other information considered to be essential and when there was consensus the additional fields were added. This process resulted in the form shown. Figure 5 - FNC BPO Form We know the score. www.collateraldna.com 16
Score Development To develop a BPO Score, the same process was used that was used for the development of the Appraisal Score. Initially, weights were assigned to the rules based on expert opinion of the seriousness of the rules. Then, as BPOs were received from clients identifying both good and bad BPO reports, statistical development of the score was used to define the appropriate parameters. The same equations for the function and for the optimization shown in equations 1 and 2 above were used. Monitoring of the score and re-estimation of the parameters is conducted as appropriate for the maintenance of the performance of the BPO Score. Score Reporting The Appraisal Score and the BPO Score are typically delivered in the reports shown in figure 6. These reports provide information on the subject property, they show a picture of the property if it can be extracted from the appraisal or BPO. The Appraisal Score provides an indicator of the degree to which the appraisal is compliant and shows the score. There are no consistent standards for BPO compliance so that component is not on the BPO Score Report. Following that information is the list of all of the rules that have fired. The rules are shown in the order of compliance rules (appraisal), risk rules, external data validation rules and market condition report (appraisal) rules. Within each section, the rules are shown in an order that is consistent with the order of the data within the appraisal or BPO that caused the rule to fire. We know the score. www.collateraldna.com 17
Figure 6 - Appraisal Score and BPO Score Reports We know the score. www.collateraldna.com 18
Appendix - Example Test The information below describes the distribution of the Appraisal Score based on actual production data. An FNC client conducted the test and permitted the results to be shared. The production data is aggregated across many lenders, thousands of appraisers and a wide and extensive distribution of geographical locations including all major metropolitan markets. The appraisals with valuation problems were identified by the client. Appraisal Score To evaluate the performance of the score, 20,027 recent new form 1004 appraisals were pulled from production. These were paired with 88 bad appraisals provided by an FNC CMS client. These bad appraisals were determined to have significant valuation problems after a detailed review by a panel of appraisal experts at the client. All data was extracted from the 1004 appraisals and run through the GAAR 4.1 rule set. The first graph shows the distributions for the scores with the production appraisals shown in blue and the appraisals identified as bad shown in brown. While the second graph shows the same data presented as cumulative distributions. Figure 7 - Distributions for Bad and Production Appraisals We know the score. www.collateraldna.com 19
Figure 8 - Cumulative Distributions for Bad and Production Appraisals The following tables show the distributions of the scores by decile in the first table and by centile for the lowest 20% of the scores in the second table. We know the score. www.collateraldna.com 20
Figure 9 - Percent of Bads within each decile We know the score. www.collateraldna.com 21
Figure 10 - Separation of the two distributions Figure 11 - Separation Gain We know the score. www.collateraldna.com 22
Figure 12 - Distribution of Bads by Centile Figure 13 - Difference by Centile We know the score. www.collateraldna.com 23
Figure 14 - Gain Distribution The Appraisal Score is currently in production most major origination lenders. The distribution of the scores is monitored weekly and these distributions have been found to remain very consistent. It is clear that there is a fairly significant difference in the distributions of the known bad appraisals relative to the production appraisals and QA analysis of the performance has shown that to remain consistent. We know the score. www.collateraldna.com 24