Lessons Learned on Cooperative Government/Industry Appraisas aka Registered Appraisas Meanie G. Benhoff Integrity Appications, Inc mbenhoff@integrity-apps.com
Cooperative Appraisas A definition: Government members or representatives participate on a corporate assessment Goa is to estabish additiona confidence in impartiaity and objectivity of assessment resuts Resuts signed by government members to assert that objective appraisa process was used in conformance with the instrument s method description Resuts (incuding findings) registered with SEI Registration certificate confirms fact of and receipt of appraisa materias / detais / resuts and compiance with defined appraisa process May be usefu in ieu of customer conducting their own evauation of the appraised organization
Impetus for Initia Interest in Cooperative Appraisas OSD Poicy, Jan 2001, requiring Leve 3 Evauation in order to compete for DoD Acquisitions Not corporate assessment, but government (or representative) evauation ACAT1 programs, but some services appying poicy to other programs Discomfort/unwiingness to rey soey on corporate appraisas for understanding corporate capabiities & process maturity Resources and schedue impications on government evauations during source seections Program Offices coud accommodate best practices / poicy infuences through increased coaboration in corporate assessments for process improvement
Coaborative or Registered Appraisas Appraisers representing government offices participate on corporate assessments as equa member of appraisa team Trained/quaified appraisers suppied and sponsored by government office Resuts registered with SEI Contractor can make registered resuts avaiabe to prospective customers/government offices Coud be used in ieu of SCE-ike evauation for acquisition whie retaining objective perspective of appraisers not sponsored by corporate organization
Who can be a Government Member of Cooperative Appraisa Team? Government empoyee Program office member DCAA rep DCMA rep Other FFRDC CAAS/SETA Support to Program Offices or Agencies As ong as no consuting reationship to appraised organization for process improvement impementation Key Criteria Proper training and experience Participation Sponsored By (Paid for By) Government Agency
Roe of Government Representatives on Appraisa Team Understand corporate objectives for appraisa Bring experience / appraisa knowedge/ mode knowedge as fu-fedged member of appraisa team Fufi responsibiities as fu-fedged appraisa team member Not merey an observer of the appraisa team Ensure their vote/voice counts as much as every other appraisa team member After appraisa: Sign registered appraisa forms Attesting to competeness/vaidity of process used for appraisa Respond to questions from prospective consumers of appraisa information during next 2 years Government program offices seeking maturity eve information in support of acquisition
1 st Registered Appraisa -- Context First cooperative appraisa conducted Summer 02 Appraisa Method: SCAMPI V 1.1 Reference Mode: CMMI SE/SW, Staged, Leve 5 Scope of appraisa, Lockheed Martin, M&DS Size of team: 6 3 of the 6 were SEI-authorized ead appraisers 2 of the 6 were SCAMPI ead assessors On Site Window: 3 days team training/readiness review 10 days of on-site appraisa activities
Factors Affecting Effectiveness of Cooperative Appraisa Eary identification and invovement of Government appraisa team members Panning Quaifications of team members Composition/Responsibiities of mini teams Interpersona dynamics of appraisa team members Readiness of the appraised organization
Lessons Learned 1 Eary Identification/Acceptance of Government Appraisa Team Members (6 months or onger before appraisa) Ensure entire appraisa team is baanced/optimized Wi drive appraisa team approach Match mini teams to compement experience/expertise of a appraisa team members Organizationa overviews and documentation needs Aows for optimized PA assignments Preserves appraisa schedue with eary ock-in Aows time to identify and resove any training needs Aows time to ook for aternatives if nominee acking critica training/experience
Lessons Learned 2 Effective Panning Invove government-sponsored appraisa team members AS EARLY AS POSSIBLE in corporate panning activities Heps buid shared understanding of corporate objectives and expectations - Senior management s focus on process improvement and maturity eve rating - Historica background of organization in their process journey (what has worked, what hasn t) Address team buiding, training, appraisa focus issues eary without disruption to corporate assessment pans
Lessons Learned 3 Quaifications of Government Team Members Must have strong experience with forma appraisas Must be a ead appraiser or candidate ead appraiser Must have strong foundation with reference mode Experience using same mode in appraisas Must have ampe experience with reevant deveopment & engineering activities 10-15 years system deveopment Simiar business/technica domain a pus Government Members represents credibiity of appraisa to other Government Agencies v the credibiity of their affirmation is imited by their credibiity as an appraiser
Lessons Learned 4 Composition/Responsibiities of Mini Teams SCAMPI concept of mini-teams does not directy support concept that government members of cooperative assessments can attest to process and be comfortabe with resuts Mini-team activities can be much more diffused than in CBA-IPI or SCE - Aows for more in-parae data gathering and consoidation Make sure Appraisa Pan aows for sufficient in seria data gathering and processing to accommodate Registered Appraisa objectives Make sure team data consoidation and consensus activities aow for sufficient time to share information across mini-team - Mini Teams responsibe for justifying characterizations at project eve to rest of the team during consensus Not just counting types/pieces of objective evidence - Red-teaming project characterizations across mini-teams in preparation for team consensus
Lessons Learned 4 (continued) Composition/Responsibiities of Mini Teams Put considerabe thought into how to organize mini teams given participation of government representatives Most controversia PA s wi be those at higher maturity eves Government members wi have ess famiiarity with organizationa aspects of processes Don t put government members on same mini team Don t put government members ony on ess controversia or ess stringent PA s Don t organize mini teams by maturity eve Doesn t baance work across mini teams Consider organizing mini teams by process category or some other method to baance appraisa work by a conscious theme Project Mgmt (6) Engineering (6) Process Mgmt (5) Support (5)
Lessons Learned 5 Interpersona Dynamics of Appraisa Team Members High probabiity government members have not been on an appraisa with rest of team members before High probabiity government members not as famiiar with organization s poicies, standards, processes, terminoogy, etc as rest of team (which more than ikey wi have experience appraising this organization) Team buiding and team communication is crucia to successfu appraisa Make time for these tasks during panning and training activities Mode interpretations need to be normaized across team Even with team of we-quaified, experienced evauators Objective evidence interpretations and definitions of sufficiency need to be consistent and reasonabe What s a Direct Artifact versus Indirect Artifact versus Direct Affirmation? What kind of objective evidence is sufficient to demonstrate fuy impemented? - One direct artifact (i.e. minutes from one meeting)? There are many types of direct artifacts so what wi be sufficient
Lessons Learned 6 Readiness/Maturity of the Appraised Organization Meeting the intent of the mode as we as the etter of the aw Conservative Mapping of Organization/Project Processes and Artifacts to Mode Organization doesn t try to stretch processes to appy to higher eve process areas Avaiabiity of additiona objective evidence and peope to respond to appraisers questions May be more questions/info requests than in typica corporate assessment Organization wecomes an objective appraisa
Output of Registered Appraisa* Statement Statement of of Appraisa Appraisa Resuts Resuts Appraisa Appraisa Findings Findings Outbrief Outbrief Characterization Characterization of of Organization Organization by by PA PA Significant Significant Strengths Strengths and and Weaknesses Weaknesses Organization/Division Organization/Division Projects Projects Appraised Appraised Appraisa Appraisa Mode Mode Appraisa Appraisa Method Method Signatures Signatures Sponsor Sponsor Lead Lead Appraiser Appraiser Government Government Reps Reps (& (& contact contact info) info) Registered resuts vaid for 2 years SEI Repository for Registered Appraisa Resuts *For further information contact SEI Customer Reations at 412-268-5800 or customer reations@sei.cmu.edu
Remaining Poicy Issues Degree to which registered appraisas used in source seections Education/awareness/motivation FAR impications for competitions If not a offerors in acquisition have cooperative appraisa resuts avaiabe/registered Near term staffing drain on government agencies to get initia cooperative appraisas registered Rey on FFRDCs and CAAS/SETA
Summary Age-od question: Does sponsorship and appraisa team composition affect outcome/resuts of appraisa? Age-od constraints: Staffing/resource constraints for impementing OSD poicy Impact of Government Cass A appraisas on acquisition schedues Soution sets: Other than SCAMPI Cass A Appraisas SCAMPI Cass B Appraisa Evauation Method (to be defined eary 03) System / Software Risk Evauations Process Benchmarking Evauations Cooperative Government/Industry Appraisas with Registered Resuts