Report for PROPERTY VALUATION SERVICES CORPORATION IPTI CERTIFICATION REPORT. December 7, 2015

Transcription

1 Report for PROPERTY VALUATION SERVICES CORPORATION IPTI CERTIFICATION REPORT December 7, 2015 Institute international de la fiscalité immobilière Suite 2308, 4950 Yonge Street Toronto, Ontario, Canada M2N 6K1 Tel: Fax:

2 Kathy Gillis Chief Executive Officer Park Place II, Building A, Suite Brownlow Avenue Dartmouth, NS B3B 2B4 December 7, 2015 Dear Ms. Gillis, Re: PVSC Audit Review IPTI Certification The (IPTI) has completed its certification review of the Property Valuation Services Corporation (PVSC), Nova Scotia, in terms of its appraisal outcomes and business processes and procedures. IPTI examined the following areas of PVSC s core valuation business: 1. Statistical Review of Assessment Outcomes 2. Policy and Procedure Review 3. Statutory and Communication Review 4. Special/Unique Properties Review 5. Quality Framework/Business Improvement Review As the worlds leading organization on property tax policy and practice, IPTI is well qualified to evaluate and make meaningful conclusions regarding assessment jurisdictions worldwide. IPTI used the International Association of Assessing Officers (IAAO) standards in making these judgements along with its considerable experience in the field of mass appraisal. It is IPTI s professional opinion, based on its expertise and objective evaluation in terms of best practices for assessment jurisdictions that PVSC has achieved what IPTI considers to be a very good level of certification, achieving a total score of 89.5% out of a possible 100%. It is IPTI s professional opinion that PVSC has been compliant in terms of property valuation as mandated by s. 42(1) of the Nova Scotia Assessment Act, RS, c23. s1 (as amended) and with IAAO standards for mass appraisal. Based on its review, IPTI has confidence in PVSC s ability to produce and provide a quality assessment roll for municipal units throughout Nova Scotia and for individual property owners in the Province. Ruel J. Williamson Ruel Williamson Chief Operating Officer Institute international de la fiscalité immobilière Suite 2308, 4950 Yonge Street Toronto, Ontario, Canada M2N 6K1 Tel: Fax: ipti@ipti.org

3 P V S C C e r t i f i c a t i o n P a g e 2 Table of Contents Introduction... 4 Executive Summary... 5 SWOT Analysis Section 1 Statistical Review and Assessment Outcomes Statistical Review & Assessment Outcome SWOT Analysis...13 Ratio Analysis Design...14 Sales Sample Representativeness...16 Sales Dataset Creation and Refinement...17 Determine Appropriate Stratification...19 Determine Time Trends to Bring Sales to Assessment Date...21 Outlier Identification and Remediation...22 Summary Measures for Assessed Value, Sale Price, and Ratio...24 Appraisal Level...25 Appraisal Uniformity...28 Detecting Selective Reappraisal...31 Nature of Changes from Formal Appeals...34 Section 2 Policy & Procedures Review Policy and Procedures SWOT Analysis...37 Parcel Updates...38 Valuation Process Timeline...39 Cost Method...40 Income Method...41 Sales Comparison Method...43 Market Modeling...44 Valuation Methodology Choices...47 Land Valuation...49 Model Stratification...50 Accounting for Location...51 Application of Models to All Properties...53 Valuation Reports...54 Selective Reappraisal...56 Value Defense...58 Sales Validation...59 Data Collection Practices...61 Data Query Checks...63 Data Entry Quality Control...65 Plan for Data Quality Review...66 Legal Framework...67 Human Resources...69 Appeals Process...71 Property Tax and CAMA System...72 Section 3 Statutory and Communication Review Statutory and Communications SWOT Analysis...75 Annual Report...76 Public Relations...76

4 P V S C C e r t i f i c a t i o n P a g e 3 Public Speaking Engagements...79 Transparency to Data...79 Spatial Representation of Data...81 Special Purpose Data...82 Section 4 Special & Unique Property Special & Unique Properties SWOT Analysis...85 Tracking Special and Unique Properties...86 Guides for Properties...87 Valuation Professionals/Contracting...88 Roll Handling Procedures...89 Quality Control...90 Section 5 Quality Framework/Business Improvement Quality Framework/Business Improvement SWOT Analysis...92 Quality Improvement Plan...93 Plan Assessment...94 Plan Approach...95 Quality Membership...96 Input and Feedback...97 IPTI Certification Documentation...98 Addendum A - PVSC Management Responses Addendum B Curriculum Vitae - Ruel Williamson, MBA

5 P V S C C e r t i f i c a t i o n P a g e 4 Introduction BACKGROUND In an effort to produce and maintain a world class, quality assessment product, PVSC entered into a formal agreement with the International Property Tax institute (IPTI) to attain an objective certification of PVSC s core business. IPTI is an independent, not-for-profit organization that has achieved international recognition for its expertise in areas of property taxation and assessment practice. IPTI assists governments, organizations, corporations, taxpayers and professionals alike to better understand the complex issue of financing local governments, and in particular, the essential role property taxation has in this process. IPTI s mission is to provide impartial, objective expert advice in the area of property tax systems and promote the concept that these systems should be fair and equitable and meet the needs of all stakeholders, i.e. governments, taxpayers, practitioners and academics. The merits of attaining IPTI Certification are: Increased public trust in the assessment process Compliance with designated standards and statutes Improved business practices International recognition in terms of quality and continued oversight A road map for continual improvement of processes, practices and procedures Greater professional development for staff IPTI certification will ensure a path of continuous improvement for PVSC in terms of its core business. It helps to evaluate and measure PVSC s continued compliance with legislated requirements by reviewing and updating internal and external audit practices currently in place. The certification project will also aid PVSC in several operational areas. It provides PVSC with a set of best practices by which to measure the quality of its assessment administration. It also provides a framework to fairly and consistently use as a measurement tool for the Corporation. A certification score of 80 or more points out of a possible 100 is regarded as good. Jurisdictions which score between 90 and 95 points are regarded as very good. Jurisdictions which score above 95 points are considered excellent THE PROCESS To achieve certification status, IPTI examined PVSC across five main areas. These were: 1. Statistical Review and Assessment Outcomes 2. Policy and Procedure review 3. Statutory and Communication review 4. Special/Unique Property Review 5. Quality Framework/Business Improvement A major step in the process of certification was a gap analysis performed by IPTI with regard to PVSC s principles, processes, and procedures in all of the above five areas. The initial review of existing documentation provided insight and direction as to the areas of the Corporation which required a greater level of detailed policies, procedures, standards and guides. In all, over one hundred thirty separate documents were provided to IPTI for review. These documents covered all of the current processes, procedures, standards and other internal correspondence deemed relevant to value production, review and appeal.

6 P V S C C e r t i f i c a t i o n P a g e 5 Executive Summary 1. The (IPTI) was commissioned by the Property Valuation Services Corporation (PVSC) of Nova Scotia to undertake a certification audit of the PVSC s operations. 2. The Nova Scotia Assessment Act and associated regulations govern the implementation of the property tax system throughout the Province. Section 42(1) of the Assessment Act, RS, c.23, s.1, as amended, states the following: All property shall be assessed at its market value, such value being the amount which in the opinion of the assessor would be paid if it were sold on the a date prescribed by the Director in the open market by a willing seller to a willing buyer, but in forming his opinion the assessor shall have regard to the assessment of other properties in the municipality so as to ensure that, subject to s.45a, taxation falls in a uniform manner upon all residential and resource property and in a uniform manner upon all commercial property in the municipality. 3. PVSC provided access to over one hundred thirty documents which were used in conducting the certification process. These documents were helpful in providing an overall understanding of the organization, along with the depth and breadth of operational details. 4. The methodology for the certification process was based on reviewing PVSC processes and outcomes against a set of best practices. This was done by reviewing fifty (50) individual processes throughout the organization across five (5) major operational areas: a. Statistical Review & Assessment Outcomes b. Policy and Procedures c. Statutory and Communication d. Special & Unique Properties e. Quality Framework/Business Improvement 5. In evaluating PVSC IPTI attributed the following weighting system: a. Statistical Review & Assessment Outcomes This is deemed the most important measure of performance within the mass appraisal profession as it independently measures the overall performance of the key organization deliverable, i.e. property values. 50% of the total score is thereby attributed to this section. b. Policy and Procedures This is deemed critical as an organizations policies and procedures guide how it conducts its daily business and ensures uniformity across the Province. 30% of the total score is thereby attributed to this section. c. Statutory and Communication This is deemed important as it outlines if the organization is communicating appropriately to both internal and external stakeholders. 9% of the total score is thereby attributed to this section. d. Special & Unique Properties This is deemed important as special and unique properties are often high-profile and can have a significant tax impact on an individual municipality. Ensuring that these properties are handled correctly is an important component to the overall assessment process. 6% of the total score is thereby attributed to this section. e. Quality Framework/Business Improvement This section is deemed important as it measures if the organization is continually looking at not only the quality of its work but also its review of business processes and improvement. An organization that does not constantly seek to improve will typically regress over time. 5% of the total score is thereby attributed to this section.

7 P V S C C e r t i f i c a t i o n P a g e 6 6. PVSC scored 89.5 out of 100 total points in the certification process. This score implies that the organization is operating at a high level (good) when compared against industry best practices. The chart below shows how scoring from the five major operational areas are broken down: The percentage of total score that is covered by each operational area is indicated below: Category Potential Score Actual Score Statistical Review & Assessment Outcomes 50% 45 Policies & Procedures 30% 26 Statutory & Communications 9% 8 Special & Unique Properties 6% 5.5 Quality Framework / Business Improvement 5% 5 Total Score: 100% In IPTI s opinion, we determined that PVSC is in compliance, in all material aspects, with the Statistical Review and Assessment Outcomes section of the certification process. 11 review criteria (see below) were evaluated with a statistical review of values broken down into two main statistical categories. These two categories are Appraisal Level and Appraisal Uniformity. The following are key findings: a. PVSC materially complies with the IAAO standard on ratio studies. b. PVSC publishes annually a roll quality report outlining the key statistical review(s) undertaken by the Corporation. c. PVSC can enhance its roll quality report by adding time-trending to its sales analysis. d. PVSC can enhance its roll quality report by testing applicable statistics with a confidence interval.

8 P V S C C e r t i f i c a t i o n P a g e 7 Review Criteria and Scores Item Score Ratio Analysis Design 5 out of 5 Sales Sample Representativeness 2 out of 2 Sales dataset creation and refinement 2 out of 2 Appropriate Stratification 2 out of 2 Time Trends 0 out of 2 Outlier Identification and Remediation 2 out of 2 Summary Measure for Assessed Value/Sales 5 out of 5 Appraisal Level 10 out of 11 Appraisal Uniformity 10 out of 11 Detecting Selective Reappraisal 4 out of 5 Nature of Changes from Formal Appeals 3 out of 3 TOTAL SCORE 45 of 50 IPTI has rated PVSC with a score of 45 out of 50 total points. This corresponds to a passing score in this category and shows a high degree of competence with respect to value production. 8. In IPTI s opinion, we determined that PVSC is in compliance, in all material aspects, with the Policy and Procedure Review section of the certification process. 23 review criteria (see below) were evaluated in determining the overall compliance score. The policy and procedure statements currently in effect within PVSC were compared against industry standards and international best practices. The following are our key findings: a. PVSC provided extensive documentation regarding internal policy and procedure. b. PVSC follows recommended mass appraisal practices when valuing property. c. PVSC conducts is valuation business in accordance with the rules outlined in the Nova Scotia Assessment Act. d. PVSC has shown it continually reviews and improves upon its policy and procedural documentation.

9 P V S C C e r t i f i c a t i o n P a g e 8 Review Criteria and Scores Item Score Parcel Updates 1 out of 1 Valuation Process Timeline 1 out of 1 Cost Method 2 out of 2 Income Method 2 out of 2 Sales Comparison 0 out of 2 Market Modeling 1.5 out of 2 Valuation Methodology Choices 1 out of 2 Land Valuation 2 out of 2 Model Stratification.5 out of.5 Accounting for Location 1 out of 1 Application of Models to all Property 1 out of 1 Valuation Reports 1 out of 1 Selective Reappraisal.5 out of 1 Value Defense 2 out of 2 Sales Validation 1 out of 1 Data Collection Practices 2 out of 2 Data Query Checks 1 out of 1 Data Entry Quality Control 1 out of 1 Plan for Data Quality Review.5 out of.5 Legal Framework 1 out of 1 Human Resources 1 out of 1 Appeals Process 1 out of 1 Property Tax/CAMA System 1 out of 1 TOTAL SCORE 26 out of 30 IPTI has rated PVSC with a score of 26 out of 30 total points. This corresponds to a passing score in this category and shows a high degree of competence with respect to its policies and procedures. 9. In IPTI s opinion, we determined that PVSC is in compliance, in all material aspects, with the Statutory and Communication Review section of the certification process. 6 review criteria (see below) were evaluated in determining the overall compliance score. The Statutory and Communication Review considered the Corporation s public relations plan and analyzed it against international best practices. The following are key findings: a. PVSC publishes an annual report which is transparent to the stakeholders regarding its operations, plans and policy issues. b. PVSC creates, maintains and follows an annual public relations program. c. PVSC provides transparency of its data through many channels including the My Property Report web page.

10 P V S C C e r t i f i c a t i o n P a g e 9 Review Criteria and Scores Item Score Annual Report 3 out of 3 Public Relations Program 1 out of 1 Public Speaking Engagements 1 out of 1 Transparency to Data 2 out of 2 Spatial Representation of Data.5 out of 1 Special Purpose Data.5 out of 1 TOTAL SCORE 8 out of 9 IPTI has rated PVSC with a score of 8 out of 9 total points. This corresponds to a passing score in this category and shows a high degree of competence with respect to its Statutory and Communication Review section. 10. In IPTI s opinion, we determined that PVSC is in compliance, in all material aspects, with the Special/Unique Property Review section of the certification process. 5 review criteria (see below) were evaluated in determining the overall compliance score of this section. The following are key findings: a. The CAMA system in use is sufficiently robust to manage Special/Unique property data. b. PVSC appraisal staff are trained, certified and can accept all assignments within the Province related to Special/Unique Property valuation. Review Criteria and Scores Item Score Tracking Special & Unique Property 1 out of 1 Guides for Properties.5 out of 1 Valuation Professionals/Contracting 2 out of 2 Roll Handling Procedures 1 out of 1 Quality Control 1 out of 1 TOTAL SCORE 5.5 out of 6 IPTI has rated PVSC with a score of 5.5 out of 6 total points. This corresponds to a passing score in this category and shows a high degree of competence with respect to its Special/Unique Property Review section. 11. In IPTI s opinion, we determined that PVSC is in compliance, in all material aspects, with the Quality Framework/Business Improvement section of the certification process. 5 review criteria (see below) were evaluated in determining the overall compliance score of this section. The following are key findings: a. PVSC is an active member in several organizations who have quality as a primary goal. b. PVSC regularly solicits and incorporates feedback from all of its key stakeholders.

11 P V S C C e r t i f i c a t i o n P a g e 10 Review Criteria and Scores Item Score Reassessment Plan 2 out of 2 Plan Assessment.5 out of.5 Plan Approach.5 out of.5 Quality Membership 1 out of 1 Input & Feedback 1 out of 1 TOTAL SCORE 5 out of 5 IPTI has rated PVSC with a score of 5 out of 5 total points. This corresponds to a passing score in this category and shows a high degree of competence with respect to its Quality Framework/Business Improvement 12. IPTI s certification review further found: a. PVSC obtained all of the points on forty (40) of the fifty (50) certification processes considered. b. All five major operational areas obtained qualifying to excellent scores. 13. IPTI makes a number of important recommendations for improvement. These include: a. PVSC should implement time-trending as part of the organization s annual statistical analysis. b. PVSC should implement testing statistical values using a confidence interval. c. PVSC should look at implementing the sales comparison method of valuation for residential property in localities where there are adequate sales samples. d. PVSC should incorporate more graphical representations of data from its operations to depict year over year operational changes. e. PVSC should improve valuation guidelines related to special and unique properties. f. PVSC should adopt and incorporate a formalized quality feedback mechanism into its organization. 14. IPTI will be pleased to discuss the outcome of our analysis, or any other aspect of this report, upon request.

12 P V S C C e r t i f i c a t i o n P a g e 11 SWOT Analysis SWOT stands for strengths, weaknesses, opportunities and threats. A SWOT analysis identifies strengths and weaknesses within an organization, and outside opportunities and threats. The most important parts of a SWOT analysis specify the actions that correspond to the elements identified. By using the results of the analysis to improve the situation of the organization, the likelihood of developments that negatively affect the business can be reduced while improving performance. Strengths are internal characteristics of an organization such as skills, resources, capabilities and brands. Weaknesses are internal characteristics such as sources of customer or employee dissatisfaction. Opportunities are external market possibilities, technological improvements or competitor weaknesses. Threats come from external factors such as new competitors, dissatisfied customers or changing market needs. The first step was to list all possible elements as a team, a brain-storming process. As part of the learning process of certification, IPTI outlined five key areas within PVSC which were determined to be of importance within PVSC to examine. The purpose of the SWOT analysis is to suggest actions that are appropriate for the organization to improve its situation. It should be noted that only a limited number of actions are possible at any one time, given typical limitations on resources and personnel. Normally, it is not necessary to keep more than the top two or three elements of each type to specify several of the most important courses of action the organization might take. The most important part of a SWOT analysis is to improve the viability of the organization. Important threats coupled with weaknesses typically put at risk the organization's future, and the SWOT analysis identifies these risks. The analysis pairs external threats with internal weaknesses to highlight the most serious issues faced by the organization. The organization must now decide whether it is most appropriate to eliminate the internal weakness, or reduce any external threats. It is possible to eliminate internal weaknesses by assigning organization resources to fix the problems. External threats can be reduced by abandoning the threatened area of business or meeting the threat after strengthening the business. The SWOT analysis for each of the five major areas that were analyzed for this project occurs at the beginning of each area section throughout this report.

13 P V S C C e r t i f i c a t i o n P a g e 12 Section 1 Statistical Review and Assessment Outcomes The statistical review and assessment outcomes process is, in reality, the most important of the five sections of this certification review. Assessment outcomes are simply the statistical review of the end product of PVSC (property valuations used for property tax purposes). These results are subject to comparison against international best practices and benchmarks. Item Score Ratio Analysis Design 5 out of 5 Sales Sample Representativeness 2 out of 2 Sales dataset creation and refinement 2 out of 2 Appropriate Stratification 2 out of 2 Time Trends 0 out of 2 Outlier Identification and Remediation 2 out of 2 Summary Measure for Assessed Value/Sales 5 out of 5 Appraisal Level 10 out of 11 Appraisal Uniformity 10 out of 11 Detecting Selective Reappraisal 4 out of 5 Nature of Changes from Formal Appeals 3 out of 3 TOTAL SCORE 45 of 50

14 P V S C C e r t i f i c a t i o n P a g e 13 Statistical Review & Assessment Outcome SWOT Analysis STRENGTHS Dataset Creation Stratification Outlier Identification Value Change Analysis WEAKNESSES Time Trending Testing for Representativeness OPPORTUNITIES Stratification Nature of Changes From Appeals THREATS The statistical review and assessment outcomes area tells us how well the appraisal results compare statistically to the International Association of Assessing Officers (IAAO) ratio study standards. PVSC annually produces a comprehensive Roll Quality Report that calculates the key statistics needed to determine the organization s relative equity and uniformity against the stated valuation standard. IPTI s review has pointed out two weaknesses that are part of PVSC s 2016 organizational goals to rectify. Overall, the extensive nature of the annual roll quality report allows the organization to stay informed of progress in the area of assessment results. By addressing the weaknesses, the corporation will enhance an already strong area of the business.

15 P V S C C e r t i f i c a t i o n P a g e 14 Ratio Analysis Design Justification The goal of the ratio study analysis within certification is to completely analyze the results of a valuation and use that analysis to draw conclusions. The ratio analysis is worthless, however, if it isn t well designed and if important goals in the ratio analysis are not clearly defined. It is important therefore to define a set of necessary criteria for the design and general execution of a ratio analysis plan. Dependencies IAAO Standard on Ratio Studies Timetable This objective should be met as a part of the completion of the ratio analysis. Review Criteria 1. The purpose of the ratio analysis must be clearly defined from the outset. Is the goal of the ratio analysis merely to provide oversight, or is there the possibility that equalization or a reassessment could be ordered? The purpose must be determined from the beginning. Points 1 of 1 2. As a part of the revaluation process, a ratio analysis could be conducted for four possible reasons: a. as a starting baseline for appraisal performance; b. to analyze the preliminary values to correct problems; c. to judge the quality of the values used in the assessment notices to ratepayers; and d. to judge the quality of the final values after either informal or formal appeals have concluded. 3. Ratio analyses may also be completed as needed throughout the year to solve a specific problem or answer a specific question. PVSC reviews ratio study statistics as a regular part of the annual roll quality report along with other quality items. 4. There should be a division to independently conduct ratio analyses every appraisal cycle with PVSC Executives given an in-depth explanation of the results. 1 of 1 1 of 1 1 of 1 5. Is the ratio analysis sample comprised of sales only or does it also include independent appraisals to make up for lack of sales? This must be known at the 1 of 1 outset as it affects the design of the ratio analysis. PVSC uses sales only when it conducts is ratio study. Scoring Summary 5 of 5

16 P V S C C e r t i f i c a t i o n P a g e 15 Review of Ratio Analysis Design PVSC annually publishes its Roll Quality Report. This report outlines key statistical measurements for appraisal level, uniformity and equity. The current Roll Quality Report contains all of the minimum level statistical analysis. Upgrades to the number and type of statistics have been initiated for the 2016 roll quality report. These additions go well beyond the minimum number and type of statistical calculations and review of the roll. In addition, PVSC provides a statistical breakdown by municipality where applicable sample sizes exist. The upgrades to the 2016 Roll Quality Report will provide additional insight into individual municipality statistics. Document(s) Queries and Reports Ratio Analysis Report/Roll Quality Report Metric Confirm that the ratio study conducted by PVSC has a clear indication of what is being measured by the study and what the guidelines are. PVSC Management Response

17 P V S C C e r t i f i c a t i o n P a g e 16 Sales Sample Representativeness Justification Sales samples are an observational means of collecting survey results. The nature and quantity of the group of properties that sell in a given time period is therefore outside of the control of the individual performing the analysis. This means that, because of various societal, geographical, or economic factors, the sales sample may end up being unrepresentative of the population of properties. This is potentially dangerous because the conclusions from the ratio analysis will be applied to all the properties as a whole. It is therefore important to test for sales sample representativeness so that this danger can be avoided. Dependencies IAAO Standard on Ratio Studies IAAO Standard on Automated Valuation Models (AVMs) Timetable This objective should be met as a part of the completion of the ratio analysis. Review Criteria 1. In each major stratum, sales samples should be representative of the population. The jurisdiction should run a statistical test for sample representativeness to test for this. 2. If a sample fails the test for representativeness then, as a remedial measure, simple random samples should be used by valuation range strata to insure groups are represented equally. Alternatively, the time period used for the sales in the ratio study could be lengthened in the hope that this may make the sample more representative; but this step, even if it is possible, may not work. Points.5 of.5.5 of.5 3. The test for sample representativeness must be detailed in the Ratio Analysis Report. 1 of 1 Scoring Summary 2 of 2 Review of Sales Sample Representativeness PVSC annually publishes the Roll Quality Report. This report outlines key statistical measurements for appraisal level, uniformity and equity. The ability to measure for sample representativeness is being added to the PVSC Roll Quality report for 2016 and beyond. Document(s) Queries and Reports Ratio Analysis Report/Roll Quality Report Metric Each ratio study should run a statistical test for sample representativeness as outlined in the IAAO Standard on Ratio Studies. PVSC Management Response

18 P V S C C e r t i f i c a t i o n P a g e 17 Sales Dataset Creation and Refinement Justification The basis of every ratio analysis is a proper sales sample dataset. Without a properly constructed sales sample dataset, the sales ratio analysis could be incorrect. Since the results of the ratio analysis are built upon the foundation of a quality sales data, it is imperative to refine the sales dataset to eliminate observations that should be excluded. Dependencies IAAO Standard on Ratio Studies Timetable This objective should be met as a part of the completion of the ratio analysis. Review Criteria 1. The sales sample dataset must contain, at minimum, the following fields: Property ID, Property Type, Sale Price, Sale Date, Sale Validity Code, Assessed Value. The Sales Query can then extract these fields. 2. The sales sample dataset must include only valid or qualified sales. Points 2 of 2 3. The sale(s) sample dataset must be reduced row-wise by a set of common-sense data edits such as: if any field in a row in missing, if the sale price is below 10,000, and if the assessed value is below 10,000. An effort must be made to rectify these data issues before a row is deleted. 4. Sales should be excluded from the sales sample dataset that have been altered significantly between the sale date and the assessment date. This applies also to sales that occur after the assessment date that may be used in a selective reappraisal detection analysis. 5. Sales in a stratum with a much higher sale price than every other sale in the stratum should be excluded from the sales sample dataset because they could skew the ratio analysis. 6. It is not valid to eliminate a sale from the ratio analysis simply because it does not meet expectations or match the assessed value. 7. The Ratio Analysis Report/Roll Quality Report should include a summary of all properties eliminated from the sales sample dataset and the reason they were eliminated. This data is available from the CAMA database as needed. Scoring Summary 2 of 2

19 P V S C C e r t i f i c a t i o n P a g e 18 Review of Sales Dataset Creation and Refinement In order for PVSC to produce its annual Roll and Roll Quality Report it must select all valid/qualified sales from the appropriate time period. This list must then be stratified by property class and further stratified by municipality. PVSC accomplishes this process through the use of a number of Cognos reports. Cognos generates the initial data downloads which are then quality checked to ensure that all parameters used to generate the reports are accurate. In addition, PVSC includes all the required sales data fields and analysis either in the Roll Quality Report or other applicable reports/downloads. Document(s) Queries and Reports Sales Query Ratio Analysis Report/Roll Quality Report Metric Confirm that the sales dataset includes all qualified/valid sales. Confirm that all sales removed from the study have been marked with a reason code. PVSC Management Response

20 P V S C C e r t i f i c a t i o n P a g e 19 Determine Appropriate Stratification Justification A properly conducted ratio analysis analyzes all sales together and significant sales strata broken out separately. These strata could be determined by property type, geographical region, or property data field range. This allows significant strata, which are often valued based on different methods and by different individual appraisers, to be reviewed individually. A problem could exist with the ratios in one stratum but not in another. The failure to appropriately employ stratification in a ratio analysis could lead to such a problem remaining undiscovered. Dependencies IAAO Standard on Ratio Studies Timetable This objective should be met as a part of the completion of the ratio analysis. Review Criteria Points 1. Strata should be defined by major property type category, geographic region, property data field range, or a combination of some or all of those factors. 2 of 2 2. Strata should be determined by taking into account the goal of the ratio analysis. 3. Strata should be determined by taking into account the nature of the factor or factors used to define the strata. Strata with too few sales should be either grouped together into a single stratum or, if that isn t appropriate, not included in the analysis at all. Dissimilar factor levels should not be grouped into the same strata, even if the goal is to arrive at more sales in a combined stratum. 4. Strata should be determined by taking into account the guideline of keeping at least twenty sales in a stratum, preferably much more. Strata must not be overstratified. 5. The goal should be to pre-define strata appropriately and then not to change them. This will work if strata are determined intelligently, both broadly enough to always have enough sales and narrowly enough to yield important information. 6. The results of strata identification should be added to the Ratio Analysis Report, including the strata definitions and the number of sales in each stratum. Scoring Summary 2 of 2

21 P V S C C e r t i f i c a t i o n P a g e 20 Review of Appropriate Stratification PVSC delineates its ratio study by Property Class and then further by Region and/or Municipality. For property classes across the Province there is typically a valid sample size. When stratifying by Municipality, the number of valid areas falls dramatically. PVSC meets the requirements of stratification, though IPTI advise the corporation to continually look for new ways to review data and seek improvement. Recommendations in this section include time-trending of sales in order to enlarge the sample size of municipalities and property class sales that do not include valid sample sizes. Currently PVSC includes ratio statistics for each stratum of five samples or more. IPTI recommends that PVSC not publish ratio statistics in a stratum that do not contain at a minimum 11 sales and to understand the relationship of sample size and confidence intervals. Document(s) Queries and Reports Ratio Analysis Report/Roll Quality Report Metric Ratio study must be delineated by the following criteria: o Property Class o Region For each criterion where data is not within the published ratio study guidelines then further stratification by age, size or sub-class is recommended. PVSC Management Response

22 P V S C C e r t i f i c a t i o n P a g e 21 Determine Time Trends to Bring Sales to Assessment Date Justification An analysis of ratios can be negatively affected by not accounting for time trends. Assessed values are calculated as of a certain date, but sale prices are collected for the sales dataset from a period of time. Over this period of time, the market undergoes changes that affect the sale prices. Time trends quantify these changes in the market, and they allow sale prices to be adjusted to the assessment date. Ignoring time trends means that assessment to sale price ratios would have distorted measures of assessment uniformity and a measure of assessment level that does not match the market at the assessment date. Dependencies IAAO Standard on Ratio Studies. Timetable This objective should be met as a part of the completion of the ratio analysis. Review Criteria 1. Time trends should be determined by stratum, when appropriate. The analyst must use judgment in determining at what level to calculate time trends for the adjustment of the sale prices to the assessment date. 2. Time adjustments should be made after other sale price adjustments but before statistical analysis. Points 0 of 2 3. Atypical sales can be removed prior to the time trend analysis, but then should be put back in after the time adjustments are determined for the formal outlier testing. Atypical sales can be determined in the same way as outliers (see the Outlier Identification and Remediation Section for more information). 4. Using a statistical model that regresses the sales ratio on time is one of the preferred methods used to arrive at the adjustment to bring sale prices even with the market on the assessment date. Care must be taken in this method to avoid model over-fitting. Other methods for accomplishing the same goal are an analysis of re-sales, an analysis of paired sales, or by comparing per-unit values over time in homogeneous strata. 5. If a model is implemented to determine the time trend, then it should use the sale price to assessment ratio as the dependent variable in order to gain stability and to make the adjustment of the ratios easier. The model should use various time variables such as month-of-sale, date-of-sale, quarter-of-sale, and their transformations as possible independent variables. The model used must adequately capture the time trend, as measured by goodness-of-fit statistics and diagnostic residual plots. 6. However, the time trend is determined, it must be used to determine a set of adjustments for the sale prices to bring them in line with the market on the assessment date. To accomplish this, PVSC should set the assessment date as the baseline and then determine a multiplier for the other points in time to bring them in line with the baseline. Scoring Summary 0 of 2

23 P V S C C e r t i f i c a t i o n P a g e 22 Review of Time Trends to Bring Sales to Assessment Date PVSC has recently re-designed the Roll Quality Report to incorporate a time-trend to all valid sales used within the report starting in While trending sales has not been used in the past, the market movement has been minimal over the past several years. This change in the statistics used within the Roll Quality Report will allow for better analysis while following industry best practices. This section of the analysis will resolve itself after the 2016 Roll Quality Report implements the time trending analysis. Document(s) Queries and Reports Ratio Analysis Report/Roll Quality Report Metric For each property class or stratification, the ratio study should have a time-trend applied to the valid sales used in the report. PVSC Management Response Outlier Identification and Remediation Justification Outliers are observations far off from what is typical. In a ratio distribution, outliers are the ratios that are at the extreme tails of the distribution. Outliers can be due to chance but, more often, they are due to a data erro r or an unknown issue. Outliers must be dealt with in an appropriate fashion lest they conceal the true results of a ratio analysis. If outliers are not accounted for, then they could skew the overall results. Dependencies IAAO Standard on Ratio Studies Timetable This objective should be met as a part of the completion of the ratio analysis. Review Criteria 1. The outlier analysis can be conducted at the stratum-level, if appropriate. 2. The 3*IQR trimming method should be used to detect extreme outliers. The 3*IQR method excludes all ratios that fall outside of the window [Q1-3*IQR, Q3 + 3*IQR], where Q1 is the first quartile of the ratios, Q3 is the third quartile of the ratios, and IQR is the Interquartile Range calculated by Q3 - Q1. This outlier trimming method is described in Appendix B: Outlier Trimming Guidelines of the IAAO Standard on Ratio Studies. Additionally, due to the fact that ratio distributions tend to be approximately lognormal, the natural logarithm of the ratios should be used rather than the ratios themselves Points 2 of 2

24 P V S C C e r t i f i c a t i o n P a g e The sales that are identified as outliers should be removed row-wise from the sales dataset unless their outlier status can be corrected by fixing a data mistake 4. The 3*IQR trimming process should only be conducted once, not multiple times. 5. The outlier analysis should be added to the Ratio Analysis Report, including the outlier trimming bounds and an identification of which sales were deleted Scoring Summary 2 of 2 Review of Outlier Identification and Remediation PVSC uses 3 times the inter-quartile range (3*IQR) as the standard way to identify outliers in the Roll Quality Report. This is the accepted IAAO standard and is employed correctly. Document(s) Queries and Reports Roll Quality Report Metric Outliers should be identified and removed using a standard statistical technique like 3 IQR. o These outliers should be included as part of the ratio study/roll quality report. PVSC Management Response

25 P V S C C e r t i f i c a t i o n P a g e 24 Summary Measures for Assessed Value, Sale Price, and Ratio Justification Summary measures are a good way to view the distribution of a set of data. Summary measures can be viewed in tabular format as summary statistics and in graphical format as summary plots. Considering the assessment to sale price ratio in particular, summary statistics allow the analyst to see the minimum ratio and the maximum ratio, which give the range of the ratio distribution, and the first and third quartiles, which can be used to extract the Interquartile Range. Both the Range and the Interquartile Range are common measures of distributional spread. The median and the mean can be quickly compared to get an idea of not just the measure of center of the ratio distribution, but an idea of the amount of scenes present of that distribution. Summary plots are useful to view the overall shape of the distribution. Summary measures also provide a window into data problems that may exist. If a mistake was made in either the sales sample refinements phase or the outlier elimination phase, then it can often be discovered by a quick analysis of the summary statistics and the summary plots. Dependencies IAAO Standard on Ratio Studies Timetable This objective should be met as a part of the completion of the ratio analysis. Review Criteria 1. The summary measures must be viewed in both a tabular and graphical format. 2. The summary measures must be viewed for the assessed value, the sale price, and the assessment to sale price ratio. Points 5 of 5 3. The summary measures must be produced for each stratum being considered in the ratio analysis, as well as overall for the entire sales sample dataset. 4. In the tabular format, the summary measures should include the minimum, the median, the mean, the maximum, and some percentiles of the given distribution, such as the first quartile and the third quartile. 5. In the graphical format, the summary plots should be ones that show the shape and center of the distribution, such as a histogram or a box-plot. 6. Conclusions should be drawn from the summary measures about the center, spread, and scenes of each distribution. 7. Conclusions should be drawn from the summary measures about the general success of the sales sample refinements and the outlier eliminations. Scoring Summary 5 of 5

26 P V S C C e r t i f i c a t i o n P a g e 25 Review of Measures for Assessed Value, Sales Price & Ratio PVSC incorporates all of the base summary measures into its Roll Quality Report. Included in these are Minimum and Maximum Ratio along with quartile ranges. Summary plots of statistics are being added to the report for Document(s) Queries and Reports Roll Quality Report Metric Summary measures should be produced for both overall numbers and the stratification of data within the study. Summary measures should include: o Minimum ratio o Maximum ratio o First and third quartiles used to extract the Interquartile range Summary plots are useful to view the overall shape of the distribution. PVSC Management Response Appraisal Level Justification A measure of appraisal level is a measure of the center of the distribution of the assessment to sale price ratio. The appraisal level indicates the percent of market value that the properties are being assessed at on the average. An appraisal level under 100% indicates that properties are being under assessed, and an appraisal level over 100% indicates that properties are being over-assessed. It is important that appraisal level be measured so that the analyst can determine whether the properties in the overall sales sample dataset and in each stratum are being assessed appropriately. A measure of appraisal level, like any statistic, encompasses both a point estimate and a confidence interval, and both should be included when that measure of appraisal level is calculated. The point estimate gives the best indication of the measure of appraisal level and the confidence interval gives a measure of the reliability of the point estimate. Both should be included to get a full picture of the appraisal level. The analysis of appraisal level is important because an equalization or reappraisal may need to be ordered if the appraisal level is unacceptable. Dependencies IAAO Standard on Ratio Studies Timetable This objective should be met as a part of the completion of the ratio analysis.

27 P V S C C e r t i f i c a t i o n P a g e 26 Review Criteria Points 1. The point estimate of the median assessment to sale price ratio must be calculated for the overall jurisdiction and each stratum. 10 of A two-sided confidence interval for the median assessment to sale price ratio must be calculated using a 95% confidence level. The confidence interval for the median can be calculated by the method of bootstrapping or one of many acceptable formulas. For small samples, the IAAO recommends using the formula in Appendix C of the IAAO Standard on Ratio Studies. For larger samples, there is multiple acceptable formulae available, including the one on Pages of the IAAO book Fundamentals of Mass Appraisal. 3. The median assessment to sale price ratio should be as near to 100% as possible. 4. The median for each stratum, as well as the overall jurisdiction, should be tested to see if it is within the range of 90% to 110%. If the median can be shown to be outside the range of 90% and 110% by a statistically significant margin, then that group of properties is said to be missing the standard and a direct equalization or reappraisal should be ordered for that group of properties. 5. If the point estimate for the median is inside the 90% to 110% range, then it is immediately said to be acceptable and it is not necessary to run the statistical test. 6. If the point estimate for the median is outside the 90% to 110% range however, then a statistical test must be conducted to see if it is outside of the range by a statistically significant amount. The statistical test can take the form of a 95% one-sided confidence interval for the median. 7. If the median point estimate is less than 90%, then a 95% one-sided upper confidence interval is used. If the median point estimate is greater than 110%, then a 95% one-sided lower confidence interval is used. In both cases, if the confidence interval overlaps the range of 90% to 110%, then the median is said to be acceptable. In both cases, however, if the confidence interval does not overlap the range of 90% to 110%, then the median is said to not be acceptable and either a direct equalization or a reappraisal is ordered for that group of properties. For ease of understanding and calculation, the needed bounds of an upper and lower 95% one-sided confidence interval are equivalent to the upper and lower bound of a 90% two-sided confidence interval. 8. If it is the case that there are a small number of sales or there is high variability in a given stratum, then it is likely the case that the confidence interval is so wide that meaningful conclusions to the statistical tests are difficult to make if the point estimate falls outside of the acceptable range. In this case, it is important to first re-examine the formulation of the stratum. If the stratum was defined too narrowly, then redefining the strata may be able to increase the number of sales enough to narrow the confidence interval considerably. If the stratum was defined justifiably, then there are two remedial measures that can be taken. 9. The first remedial measure is that the level of confidence can be reduced by 5% each year for five years from 95% to 70%, if each year the point estimate continues to fall outside of the acceptable range but the confidence interval overlaps the acceptable range. This effectively acts to progressively tighten the interval and make it easier for the statistical test to detect a statistically significant difference.

28 P V S C C e r t i f i c a t i o n P a g e The second remedial measure is that past ratio analysis cycles can be reviewed to see if the point estimate for the median was outside of the acceptable range in the same direction for five years in a row. If the point estimate was outside the acceptable range in the same direction for the five previous years in a row, then it can be concluded that the median misses the standard 11. The analysis of appraisal level should be added to the Ratio Analysis Report. Scoring Summary 10 of 11 Review of Appraisal Level PVSC incorporates all of the appraisal level statistics into its Roll Quality Report. Included in these statistics are the Mean, Median and Weighted Mean ratios for each property class. Stratification of these statistics is further refined by Municipality. Document(s) Queries and Reports Roll Quality Report Metric Mean, Median and Weighted Median ratio for each property class and stratification should fall within the published ratio study guidelines (IAAO Standards are published in the Appendix). PVSC Management Response