etrac Guide to Enhanced Appraiser Scoring Version: 2.5.0 Published: 09/22/2014 Global DMS, 1555 Bustard Road, Suite 300, Lansdale, PA 19446 2014, All Rights Reserved.
Table of Contents Introduction to Enhanced Appraiser Scoring... 3 Accessing Enhanced Appraiser Score... 4 Configuring Enhanced Appraiser Scoring... 5 Calculations for Enhanced Appraiser Scoring... 8 The Communication Score... 8 The Turn Time Score... 9 The Revision Count Score... 11 The MARS Score... 11 The Reviewer Score... 12 Reviewer Score Reporting... 13 Entering Review Score... 14 Appraiser Quality Scoring... 16 Quality of Appraisal Score... 17 Customer Service Score... 17 Complimentary Appraiser Scoring Apps... 17 Testing Enhanced Appraiser Scoring... 18 Global DMS etrac Guide to Enhanced Appraiser Scoring 2
Introduction to Enhanced Appraiser Scoring Enhanced Appraiser Scoring provides enhanced Vendor scoring to ensure the highest level of quality by monitoring turntimes, communications, revisions, and automated and manual technical reviews. The etrac appraiser scoring model consists of 5 different criteria, all of which can be weighted differently based on importance. There are several advanced parameters that can be used in the scoring of Vendors based on their turnaround time, revision count, reviewer score, and MARS score. When the Scoring options are configured and saved, the Vendor Score that displays in the Find Vendor selection screen will displays the average score for that specific zip code on orders completed over the last 6 months. The Scoring is not retroactive. The Scoring will begin at the time that the configuration is saved. Global DMS etrac Guide to Enhanced Appraiser Scoring 3
Accessing Enhanced Appraiser Score If Enhanced User Permissions are not on the System, then all Administrator Staff Users will be able to access the Enhanced Appraiser Scoring configuration page. If Enhanced User Permissions are present, Staff User access to File Delivery Configuration can be controlled. Staff User settings can be accessed from Setup, and Manage Users. Click on Edit adjacent to a Staff User. From the Staff User Profile page, locate the Enhanced User Permissions link. Click on Click here for enhanced User Permissions to access the specialized permissions for the Staff User. The specific setting for Configure Appraiser Scoring can be found within the Setup section. To allow a User to access Configure Appraiser Scoring, the option must be selected on the Staff User Profile. If the checkbox is deselected, then Configure Appraiser Scoring will no longer be available to the Staff User. Save any changes to the Enhanced User Permissions by clicking on the Save button. Global DMS etrac Guide to Enhanced Appraiser Scoring 4
Configuring Enhanced Appraiser Scoring To configure Enhanced Appraiser Scoring, click on Setup, and select Appraiser Scoring. The Enhanced Appraiser Scoring configuration page will appear. Global DMS etrac Guide to Enhanced Appraiser Scoring 5
There are five (5) configurable scoring parameters for scoring Appraisers: Turn Time Revision Count Communication Score Reviewer Score MARS Score Additional scoring options are available with the Appraiser Quality Scoring app. Enhanced Appraiser Scoring Parameters Turn Time Revision Count Communication Score Reviewer Score MARS Review Turn time is determined by Assigned to In Review or Inspected to In Review, based on what the office calculates the turn time by. One of these options must be selected. The Vendor will then be scored upon the turn time of status update to status update. This is determined by the number of times the order is sent back for revisions. Revision Count is status driven, which means that the Scoring will count how many times the orders were in the Request for Revisions/Addenda status. This parameter measures how frequently Vendors update the System. These updates notify the clients and staff that are associated with the order. This will work if the status updated in the Vendor Module, or the Staff Module. The Communication Score is based upon 4 statuses. The scoring for these statuses cannot be changed or modified. These are the order statuses that are used in the Communication Score: Accepted Scheduled Inspected In Review This is an internal review score that internal reviewers can rate/score the vendors based on the complexity of the property and the job. Reviewer scores are added directly to the order. This score can be applied to the order within the Staff Module, or the Reviewer Module. MARS is an etrac automated review tool, which can run a review when the Vendor uploads a file to the system. MARS Rules can be configured separately. The MARS Review score can be used as a parameter in Appraiser Scoring. Enhanced Appraiser Scoring determines an average of the MARS scores. Global DMS etrac Guide to Enhanced Appraiser Scoring 6
For each parameter, enter the desired weight. The Appraiser Scoring parameters must equate to 100. You may weigh the parameters in any configuration that suits your business needs. For example, if you are mainly concerned with revisions and the automated review score, each of these parameters would be configured to 50. Click the Save button to save the Scoring configuration. The Appraiser Scoring will immediately begin calculations on all order updates and status changes moving forward. Global DMS etrac Guide to Enhanced Appraiser Scoring 7
Calculations for Enhanced Appraiser Scoring The Communication Score Communication Score can track when an appraiser updates the status of orders with the minimum required status updates. Only status changes to the following statuses will be measured: Accepted by Vendor Scheduled Inspected In Review Changes to these statuses are measured, whether the update occurs in the Staff Module or Vendor Module. The total count of updates = Accepted by Vendor, Scheduled, Inspected, and In Review Status History updates. Formula Completed order count multiplied by four, then divided by count of status updates, then multiplied by 100, and subtracted by 100. 100 [(count of updates) / (completed order count * 4) * 100] Example 2 completed orders (1 order with 4 status updates) and (1 order with 0 status updates) would result in a score of 50. 100 [4 / (2 * 4) * 100] = 50 Global DMS etrac Guide to Enhanced Appraiser Scoring 8
The Turn Time Score Turn Time scoring tracks the total delivery time of the appraisal report by the appraiser. The turn time calculation has two (2) settings to choose from: Assigned to Completed Inspected to Completed This score can be confusing, as an order can be placed in the In Review status multiple times. The In Review date field on the order will be populated the first time that an order was placed in the In Review status. If the order goes into the In Review status at a later time, the original date will still populate this field. If you believe that this date is providing an unfair score for Vendors, then consider deleting the In Review date from this field prior to requesting revisions. Global DMS etrac Guide to Enhanced Appraiser Scoring 9
To remove the In Review date, click on the File Number to enter the Order Update screen. In the Status Info section, delete the date and time information, and click on the Update button to save the change to the order. If you leave this field blank, then the next time the order enters In Review status, that date and time will populate this field. Formula If the number of elapsed days is less than product turn time, then the score equals 100. Otherwise, elapsed days product turn-time is divided by product turn-time, multiplied by 100, and subtracted by 100. IF elapsed days < product turn time, THEN score = 100 - Otherwise - AVG (100 (((elapsed days product turn time) / product turn time) * 100) Example 2 completed orders (1 order with 8 elapsed days on a product with 5 day turnaround) and (1 order with 7 elapsed days on a product with 5 day turnaround) would result in a score of 50. AVG (100 (((15 10) / 10) * 100)) Global DMS etrac Guide to Enhanced Appraiser Scoring 10
The Revision Count Score The Revision Request data is compiled by using a Rev Count or Revision Count function. This function counts the number of times an order enters the Request for Revisions/Addenda status. Formula 100 - ((revision count / completed order count) * 100) Example 10 completed orders (8 orders with 0 revisions) + (1 order with 1 revision) + (1 order with 2 revisions) would result in a score of 70. (100 ((3 / 10) * 100) The MARS Score The MARS option provides a technical automated review of the appraisal report that includes rules for completeness and correctness based on USPAP, UAD, Fannie Mae, and Freddie Mac guidelines. Formula The average MARS Review Score is based on completed orders. Score is 0-100% Global DMS etrac Guide to Enhanced Appraiser Scoring 11
The Reviewer Score The Reviewer score is based upon the in-house grade of an appraisal report assigned by the AMC staff member / Internal Reviewer. When selecting an option for the Score drop-down menu, the corresponding score will be applied to the order. This information will also be reflected in the Find Vendor screen. Reviewer Score Description Score Exceeds Expectations 4 Meets Expectations / Acceptable 3 Generally Acceptable / Needs Call Back 2 Needs Improvement / Under Review 1 Not Rated 0 Below Expectations / Unacceptable -1 You can build reports based upon this information, as well. The Vendor Score that displays in the Find Vendor selection screen only displays the average score for that specific Postal Code on orders completed over the last 6 months. Formula: AVG (Review Score) for completed orders. Score is 0-4.0 Global DMS etrac Guide to Enhanced Appraiser Scoring 12
Reviewer Score Reporting You can use etrac Report Builder to create a report that takes all Vendor assignments into consideration. To build this report, use the General Report type. The field name for the numeric score is ReviewerQCRating. You can apply totals and averages on this field in the report. Typically for a report of this nature, we would use the following fields: Vendor Score Report Field Name AppraiserName FileNumber Appraiser Name DateCompleted ReviewDate ReviewerComments ReviewerDifficulty ReviewerQCRating ScoreDescription Notes Also use this field for both Sort and Group options. Order number. Full name of the Appraiser. Date that the order was completed. If the order was in this status multiple times, then this field will show the date of the original Completed status. Date that the Reviewer applied the QC score to the order. This is a text field that the Reviewer can use to enter comments or notes about their score, and their thoughts about the Appraisal Report. On the report, this will be represented as a numeric value. The numbers correspond with the drop-down menu order on the Reviewer options on the order. On the report, this will be represented as a numeric value. The numbers correspond with the drop-down menu order on the Reviewer options on the order. Also use this field for Total and Average options to determine an average Reviewer Score. On the report, this shows the corresponding verbiage to ReviewerQCRating. Report Filters DateCompleted >= RuntimeData DateCompleted <= RuntimeData The Reviewer score is determined upon the Score option that the Reviewer selects. The Difficulty options are available as an informational field, but the Difficulty score does not affect the Score. Score Vendor Score Fields and Score Difficulty Description Score Description Score Exceeds Expectations 4 Extremely Difficult 4 Meets Expectations / Acceptable 3 Difficult 3 Generally Acceptable / Needs Call Back 2 Above Average 2 Needs Improvement / Under Review 1 Average 1 Not Rated 0 Below Expectations / Unacceptable -1 Global DMS etrac Guide to Enhanced Appraiser Scoring 13
Entering Review Score Scoring can be performed from the Order Details page, and locate the Real Estate Professional Info section. Locate the Score Vendor link. The Score Vendor page includes several sections: Review Notes This field can be used to enter notes regarding how the Appraisal was scored Score A drop-down menu containing several pre-defined scoring options Difficulty A drop-down menu containing several selections for rating the complexity of the subject property. Global DMS etrac Guide to Enhanced Appraiser Scoring 14
To complete an Appraisal review, enter notes related to the Appraisal in the Review Notes field, select a review Score, and select a Difficulty rating. When selecting an option for the Score drop-down menu, the corresponding score will be applied to the order. Score Vendor Score Fields and Score Difficulty Description Score Description Score Exceeds Expectations 4 Extremely Difficult 4 Meets Expectations / Acceptable 3 Difficult 3 Generally Acceptable / Needs Call Back 2 Above Average 2 Needs Improvement / Under Review 1 Average 1 Not Rated 0 Below Expectations / Unacceptable -1 To mark the order as reviewed, click on Reviewed checkbox. Click Update to mark the order as Reviewed. Global DMS etrac Guide to Enhanced Appraiser Scoring 15
Appraiser Quality Scoring Requires Appraiser Quality Scoring App Appraiser Quality Scoring formulas score vendors based on the Quality of Appraisal and Customer Service. These additions to Vendor Scoring provide the ability to more accurately judge Vendor Panel performance and performance trends. These fields are located in the Score Vendor screen, accessible from the Score Vendor link on the order details page. The Quality of Appraisal Score and Customer Service Score fields are available for weighting in the Enhanced Appraiser Scoring Configuration Screen, as well. The Quality of Appraisal Score and Customer Service Score fields are included in the calculation of the Enhanced Appraiser Score. Requires the Enhanced Appraiser Scoring app. The Quality of Appraisal Score and Customer Service Score fields are available in the etrac Report Builder for reporting purposes. Requires etrac Report Builder app. Report Type General Report Appraisers Report Appraiser Report Card Field Name ReviewerAppraisalQualityScore ReviewerCustomerServiceScore ReviewerAppraisalQualityScore ReviewerCustomerServiceScore ReviewerAppraisalQualityScore ReviewerCustomerServiceScore Global DMS etrac Guide to Enhanced Appraiser Scoring 16
Quality of Appraisal Score The weight is based on a scale ranging from 1-100. The Quality of Appraisal field will be calculated as an average for each Vendor. Note: The Quality of Appraisal Score for an individual order can be accessed from the Score Vendor link on the Staff Module Order Form. Customer Service Score The Customer Service parameter will be located directly below the new Quality of Appraisal Score parameter and directly above the existing MARS Score parameter. The parameter Weight is based on a 1-100 total scale across all selected parameters. For example, if 20 is entered into the weight, 20% of the Enhanced Appraiser Score will be based on the Customer Service score assigned on the Score Vendor screen). Note: The Customer Service Score for an individual order can be accessed from the Score Vendor link on the Staff Module Order Form. Complimentary Appraiser Scoring Apps The Quality of Appraisal and Customer Service are also available within the Enhanced Appraiser Scoring app. Global DMS etrac Guide to Enhanced Appraiser Scoring 17
Quality of Appraisal Score For example, if 20 is entered as the weight, 20% of the Enhanced Appraiser Score will be based on the Quality of Appraisal score assigned on the Score Vendor screen. If the Quality of Appraisal field value is NULL (blank), the Vendor s average Enhanced Appraiser Score will not be impacted. This will ensure orders that are not yet scored do not negatively impact the Vendor s score. Customer Service Score For example, if 25 is entered as the weight, 25% of the Enhanced Appraiser Score will be based on the Customer Service score assigned on the Score Vendor screen. If the Customer Service field value is NULL (blank), the Vendor s average Enhanced Appraiser Score will not be impacted. This will ensure orders that are not yet scored do not negatively impact the Vendor s score. Testing Enhanced Appraiser Scoring Assign an order to a test Vendor. Log in to the Vendor Module as the test Vendor, and update the order to each status that you have selected from the Scoring parameters, allowing several minutes or longer to pass between each status update. As a Staff User, mark the order as Complete. From the Staff Module, locate the test order. Using the Find Vendor option, review the Average Score column in the Find Vendor screen. The Average Score column will display a score based on the criteria that you have configured. When compared to a separate Test Vendor that does not have any orders, you will see different values for the Average Score. Global DMS etrac Guide to Enhanced Appraiser Scoring 18