Collateral Underwriter, Regression Models, Statistics, Gambling with your License Keith Wolf, SRA, AI-RRS Kwolf Consulting Inc. Kwolf1021@gmail.com 05/20/2015
There are Lies, Damned Lies and Statistics (Mark Twain) Also known as False Positives in Collateral Underwriter or in Hard Science Hypothesis Testing, Type I errors
Collateral Underwriter UAD & CU Fail Without Data Integrity Vertical & Horizontal Inequity COD & PRD
Data Science, Statistics & Appraisers Statistics are Descriptive and Inferential Statistical Models Provide Suggested Solutions Mathematics, Statistics, Chemistry are Hard Sciences with rigid rules that can t be broken. Economics, Appraisal, Social Sciences are Soft Sciences: with rules that can be broken based on Professional Judgement. Data Science is a blend of Hard and Soft Science. Appraisers are Data Scientists that use Statistics tempered by Professional Judgement and Experience. Statistics are Probabilities with embedded Variance Analysis error rates with False Positive conclusions. Statistics DO NOT Estimate value, appraisers do.
APB Valuation Advisory #5 Identifying Comparable Properties In Automated Valuation Models for Mass Appraisal values will suffer when an important property attribute is either missing, incorrectly specified or calibrated during the modeling process. it is important for appraisers to learn about data exploration techniques, to avoid problems with missing data, and models that have been mis-specified, resulting in irrational and unexplainable calibration of a variable. estimate a value by using an iterative process of data and analysis Some off the shelf models are not iterative
Statistical Descriptive Tools Excel Pivot Table & Chart Year Built Average Price Subdivision
Statistical Descriptive Tools Tableau Table Year Built / MLS Type Type Subdivision & Price/ Design
MLS Type / Predominant Price Statistical Descriptive Tools StatWing MLS Type Area Breakdown 50% Percentages 45% 40% 35% 30% 25% 20% 15% 10% Percentage of Attached Single Percentage of Detached Single 5% 0%
Redstone: Bradford Technology Vertical Comparison
CU Provides Statistically-Derived Adjustments False Positive Appraisal Quality Progression/Regression, Define Material?
Standard 6 Mass Appraisal, Development and Reporting An Automated Valuation Model (AVM) is a mathematically based computer software program that produces an estimate of market value. Computer Assisted Mass Appraisal (CAMA) A system of appraising property, usually only certain types of real property, that incorporates computersupported statistical analyses such as multiple regression analysis and adaptive estimation procedure CU is a CAMA system with an embedded AVM
APB Valuation Advisory #5 Sample Model Structures - Direct Market Method Additive models have the form: MV = B0 + (B1 X1) + (B2 X2) + (Bxx Xxx) (Regression Coefficients) In a multiplicative model, the contribution of the variables are multiplied rather than added: MV = R0 * (X1^B1) * (X2^B2) *(Xxx^Bxx) (Rates, Multipliers, Ratios) Hybrid models are a combination of additive and multiplicative models. MV =πgq x [(πbq x Σ BA) + (πlq x Σ LA) + Σ OA] (Land & Building Components, VRA) A full discussion on the topic of model structures is available in the IAAO Standard on AVMs.
APB Valuation Advisory #5 valuation models require data representative of the full range of transaction prices and associated property characteristics in a defined area Client Supplementary Guidelines Bias Statistics through Confounding comps must bracket subjects features, be within 1 mile, 6 months, x% of GLA, Value must be within actual sale prices of comps and not above or below relies on market data from standardized property coding methods (UAD) different property coding standards produces different results The appraisers, skill, knowledge, and experience play a role in knowing which and much weight to assign an attribute/variable leading to CREDIBILITY.
Appraiser Independence? Coefficient of Dispersion can derive net and gross adjustment % from the market
CREDIBILITY The sales selected for the adjustments have attributes closest in the defined categories collectively and are placed on the adjustment grid. (REMOVE THE CONFOUNDING) Defining the metric selection is a skill in defining attribute importance and magnitude of metric selection that will improve the appraiser s impartiality from personal judgment with an objective method. Since the coefficients (adjustment rates) are derived from the transactions and have been tested using various statistical methods within the modeling process, the appraiser has presented a level of independence, impartiality, and objectivity Comparable selection must be emphasized over the adjustment process (NO MORE 10%, 15% 25% ILLUSIONARY QC RULES THAT VIOLATE APPRAISER INDEPENDENCE) the importance of finding comparable that are as similar as possible to the subject greatly reduces the need for making many (sometimes controversial) adjustments to the sales prices.
Confidence Interval A 95% confidence level is 2 Standard deviations from the mean therefore 95% of the combinations fall under the Curve 2 standard Deviations 1 Standard Deviation Statistics has a concept called the third variable, Confounding. Confounding is the third variable in a probability matrix with no easy way of segmenting out its corresponding contribution to value. ANOVA is a method for testing differences among means by analyzing variance. The difference in the mean could be the adjustment supported by paired means for the difference in Characteristics.5 Standard Deviation
Statistical Inferential Analytical Tools Statwing Redstone
Statistical Inferential Analytical Tools Excel AVT SUMMARY OUTPUT Regression Statistics Multiple R 0.973264715 R Square 0.947244205 Adjusted R Square 0.876903145 Standard Error 88407.35588 Observations 29 ANOVA df SS MS F Regression 16 1.68403E+12 1.05252E+11 13.46644771 Residual 12 93790326887 7815860574 Total 28 1.77782E+12 Coefficients Standard Error t Stat P-value Intercept -11,755,737 11807379.25-1.00 0.34 Closed Date 142 229.44 0.62 0.55 Stories 10,887 47152.96 0.23 0.82 LMT (DOM) -341 494.06-0.69 0.50 ASF 167 54.07 3.09 0.01 # Rms -19,716 25367.01-0.78 0.45 Bsmt. Beds Bsmt. Beds 115539.39 1.78 0.10 Beds 80,817 67250.38 1.20 0.25 # Full Baths 52,338 49438.11 1.06 0.31 # Half Baths -605 97721.25-0.01 1.00 Yr Blt 2,891 2569.02 1.13 0.28 Recent Rehab 28,249 48176.15 0.59 0.57 # Garage Spaces -129,504 87433.39-1.48 0.16 Lot Sq Ft 6 6.76 0.83 0.42 # Interior Fireplaces 114,581 88718.61 1.29 0.22 Basement Finish 33,916 57474.32 0.59 0.57 Subdivision -39,226 29925.12-1.31 0.21
Statistical Inferential Analytical Tools
Model Result Comparisons Excel AVT Statwing Redstone Pairs R Square 0.95 0.96 0.81 0.95 Standard Error 88,407 79,060 87,923 Observations 29 28 29 21 27 Variable Adjustment Closed Date 142 0 154 194 Stories 10,887-4,412 42,936 LMT (DOM) -341-689 455 ASF 167 193 154 158 173 # Rms -19,716-35,518 4,104 Bsmt. Beds 206,140 212,596 123,837 Beds 80,817 117,589 # Full Baths 52,338 35,466 150,583 70,891 106,313 # Half Baths -605-16,851 22,286 Yr Blt 2,891 3,728 7,515 1,505 6,205 Recent Rehab 28,249 51,582 35,520 98,617 # Garage Spaces -129,504-144,268-56,703-38,332 Lot Sq Ft 6 1 5 10 7 # Interior Fireplaces 114,581 123,775 121,658 203,149 Basement Finish 33,916 22,584 28,356 Subdivision -39,226-31,136-59,280
Case Study Subject Sale 1 Sale 2 Sale 3 Sale 4 Sale 5 Sold Pr $ 312,000 Adjustment $ 367,000 Adjustment $ 382,500 Adjustment $ 423,250 Adjustment $ 430,000 Adjustment $ Per Sq Foot 139.10 167.81 173.86 178.81 193.69 LMT 6 94 38 109 27 Closed Date 4/29/2014 1/9/2015 8/1/2014 4/30/2014 7/16/2014 # Rms 8 8 8 8 9 8 Beds 4 3 20000 4 4 4 4 # Full Baths 2 2 2 2 2 2 # Half Baths 2 1 1 1 1 1 ASF 2160 2,243 2,187 2,200 2,367-36,000 2,220 Yr Blt 1971 1977-25800 1978-30100 1978-30100 1974-12900 1979-34400 Bsmt Full Full Full Partial Partial Full Bsmt Finish Unknown Unfinished Unfinished Rec Room -9000 Unfinished Bed,den,office -28000 Lot Size 10640 6,451 29,000 10,800 10,296 11,851-8,500 11,900-8500 Garage Spaces 2 2 2 2 2 3-24000 Type 2 Stories 2 Stories 2 Stories 2 Stories 2 Stories 2 Stories Other Patio Patio Patio Screen Porch -9600 Patio Patio Condition C3 C3 C3 C3 C2-38000 C3 Net/ % 23,200 7% (30,100) -8% (48,700) -13% (57,400) -14% (94,900) -22% Gross/% 74,800 24% 30,100 $ 0.08 48,700 13% 57,400 14% 94,900 22% Adjusted Price $ 335,200 $ 336,900 $ 333,800 $ 327,850 $ 335,100 Avg Regression Adj Average Overall Price per square foot from sales comparison approach 170.66 Conditiion 38,000 See what is in RED, think we have a GLA Adjustment Error? Bsmt Finish 28,000 Land 7.00 Our R Squared says GLA is 59%, therefore should it be $103.84 and not the average of the Regression Models? Year Built 4,300 Marshall & Swift Depreciated Cost New $115.78 Avg GLA $Adj 176 Average of R square and Depreciated cost New $113.72 rounded to $114 Match Pair 1 to 2 Bed Adj 20,000 Avj Adj $ of 4 comps compared to 5 24,000 If we change the GLA adjustment to $114 per sq ft we get a new adjusted value for comp 4 at $340,252. Screen Porch Avg Adj comps compared to 3 9,600
Appraisal Uniformity Vertical Equity Comparison Method Appraisal Zestimate NAR AVM Value $335,560 $457,633 $660,780 SER/COD 11.07 15.10 21.80 A required component in ad valorem mass appraisal is demonstrating the appraisal uniformity or equitable treatment of identical or similar properties during the valuation process. A comparison method available that will reveal uniformity of values and equitable treatment is vertical equity analysis, with the caveat that the appraised value of the subject is substituted for the assessed value in the ratio analysis. This method provides users with an instant view of how the subject property is valued in comparison to other properties and how the adjustment rates affect the value
SUMMARY Appraisers are provided little guidance to help them with what the most comparable property actually means. As AVMs and CAMA become more universal, the idea of using statistical measures to evaluate comparability will be an acceptable alternative to the previous manual and subjective method. The Appraisal Industry and Mortgage Financial Services lack the education and experience to apply the methodology described; for the last 30+ years we have relied on illusionary false positives of appraisal quality using guidelines 10%, 15%, 25%, 6 months, 1 mile, 20% GLA variance Appraisers, Review Appraisers, Chief Appraisers & Underwriters all need to re-tool for these methods to become standard protocol. Many appraisers will use these tools without understanding the what, why and when associated with identifying credible results.