Key Questions Governing Evaluation

К оглавлению
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 
102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 
119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 
136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 
153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 
170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 
187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 
204 205 206 207 208 209 210 211 212 213 214 215 

To be performed effectively, evaluation for SP&M should focus on several key

questions:

1. Who will use the results?

2. How will the results be used?

3. What do the program’s clients expect from it?

4. Who is carrying out the evaluation?

The first question seeks to identify the audience. The second question seeks

to clarify what decisions will be made based on evaluation results. The third

question grounds evaluation in client expectations and program objectives.

Finally, the fourth question provides clues about appropriate evaluation techniques

based on the expertise of the chosen evaluator(s).

What Should Be Evaluated?

Some years ago, Donald Kirkpatrick developed a four-level hierarchy of training

evaluation that may be usefully modified to help conceptualize what

should be evaluated in SP&M.4

Kirkpatrick’s Hierarchy of Training Evaluation

The four levels of Kirkpatrick’s training evaluation hierarchy are reaction,

learning, behavior, and organizational outcomes or results. Reaction forms

the base of the hierarchy and is easiest to measure. It examines customer satisfaction—

that is, ‘‘How much did participants like what they learned?’’ Learning,

the second level on the hierarchy, has to do with immediate change. In

other words, ‘‘How well did participants master the information or skills they

were supposed to learn in training?’’ The third level of the hierarchy, behavior,

has to do with on-the-job application. ‘‘How much change occurred on the

job as a result of learner participation in training?’’ The highest level of Kirkpatrick’s

hierarchy, the fourth and final one, is organizational outcomes, or results.

It is also the most difficult to measure. ‘‘How much influence did the

results or effects of training have on the organization?’’

Modifying Kirkpatrick’s Hierarchy

Use Kirkpatrick’s Hierarchy of Training Evaluation to provide a conceptual

basis for evaluating an SP&M program. (Examine the Hierarchy of Succession

Planning and Management Evaluation depicted in Exhibit 13-1.)

Make the first level customer satisfaction, which corresponds to Kirkpatrick’s

reaction level. Pose the following questions:

How satisfied with the SP&M program are its chief customers?

How satisfied are its customers with each program component—such as

job descriptions, competency models, performance appraisal processes,

individual potential assessment processes, individual development

forms, and individual development activities?

How well does SP&M match up to individual career plans? How do

employees perceive SP&M?

Make the second level program progress, which is meant to correspond to

Kirkpatrick’s learning level. Pose the following questions:

How well is each part of the SP&M program working compared to stated

program objectives?

How well are individuals progressing through their developmental experiences

in preparation for future advancement into key positions?

Make the third level effective placements, which is meant to correspond to

Kirkpatrick’s behavior level. Pose these questions:

What percentage of vacancies in key positions is the organization able

to fill internally?

How quickly is the organization able to fill vacancies in key positions?

What percentage of vacancies in key positions is the organization able

to fill successfully (that is, without avoidable turnover in the first two

years in the position)?

How quickly are internal replacements for key positions able to perform

at the level required for the organization?

Exhibit 13-1. The Hierarchy of Succession Planning and Management

Evaluation

Customer Satisfaction

Organization Results

_ What percentages of vacancies in

key positions is the organization able

to fill internally?

_ How satisfied with the succession

planning program are its chief

customers?

_ How is succession planning

contributing to documentable and

measurable organization results?

_ How well is each part of the

succession planning program

working compared to its stated

objectives?

_ What organizational successes and

failures, if any, can be attributed

solely to succession planning?

_ What savings, if any, can be

demonstrated from not filling key

positions for which alternative, and

more innovative, approaches were

used to maintain equivalent results?

_ How well are individuals progressing

through their developmental

experiences in preparation for future

advancement into key positions?

_ How quickly are internal

replacements for key positions able

to perform at the level required for

the organization?

_ What percentage of vacancies in key

positions is the organization able to

fill successfully (without avoidable

turnover in the first two years in the

position)?

_ How quickly is the organization able

to fill vacancies in key positions?

_ How satisfied are targeted clients

with each program component?

_ How well does succession planning

match up to individual career plans?

Program Progress

Effective Placements

What savings, if any, can be demonstrated from not filling key positions

for which alternative, and more innovative, approaches were used to

achieve results?

Make the fourth level organizational results, which is meant to correspond

to Kirkpatrick’s outcomes or results. Direct attention to the impact of

SP&M on the organization’s ability to compete effectively, which is (admittedly)

difficult to do. Consider the following questions:

How is SP&M contributing, if at all, to documentable organizational results?

What successes or failures in organizational strategic plans, if any, can

be attributed to SP&M?

Use the guidelines in Exhibit 13-2 and the worksheet in Exhibit 13-3 to consider

ways to evaluate SP&M in an organization on each level of the hierarchy.

Of course, it is possible to evaluate an SP&M program on all four levels, and

when that is done, a scorecard for SP&M is created.

How Should Evaluation Be Conducted?

Evaluation may be conducted anecdotally, periodically, or programmatically.

Anecdotal Evaluation

Anecdotal evaluation is akin to using testimonials in evaluating training.5 It

examines the operation of the SP&M program on a case-by-case basis. As vacancies

occur in key positions, someone—often the SP&M coordinator—documents

in incident reports how they are filled. (See Exhibit 13-4 for an example

of an incident report.) The incident reports are eventually brought to the organization’s

SP&M committee for review and discussion. They provide a solid

foundation for troubleshooting problems in SP&M that the organization is

confronting. They can then be used as a basis for planning to handle similar

problems in the future.

Anecdotal evaluation dramatizes especially good and bad practices. This

draws attention to them and provides an impetus for change, a chief advantage

of the anecdotal approach. On the other hand, anecdotal evaluation suffers

from a lack of research rigor. It is not necessarily representative of typical

SP&M practices in the organization. (Indeed, it focuses on so-called special

cases, horror stories, and war stories.) It may thus draw attention to unique,

even minor, problems with SP&M in the organization.

(text continues on page 301)

Exhibit 13-2. Guidelines for Evaluating the Succession Planning and Management Program

Type/Level Purpose Strengths Weaknesses

Customer

Satisfaction

To measure client feelings

about the program and its

results.

_ Easy to measure.

_ Provides immediate feedback

on program activities

and components.

_ Subjective.

_ Provides no objective

measurement of program

results.

Program

Progress

To measure results of each

component of the succession

planning program.

_ Provides objective data on

the effectiveness of the

succession planning program.

_ Requires skill in program

evaluation.

_ Provides no measurement

of skills of benefit to the

organization.

Effective

Placements

To measure the results of the

succession decisions made.

_ Provides objective data on

impact to the job situation.

_ Requires first-rate

employee performance

appraisal system.

Organizational

Results

To measure impact of the

succession planning program

on the organization.

_ Provides objective data for

cost-benefit analysis and

organizational support.

_ Requires high level of evaluation

design skills; requires

collection of data

over a period of time.

_ Requires knowledge of the

organization’s strategy

and goals.

(continues)

Exhibit 13-2. (

continued)

Type/Level Examples Guidelines for Development

Customer

Satisfaction

_ ‘‘Happiness reports.’’

_ Informal interviews with

‘‘clients’’ at all levels.

_ Group discussion in

succession planning

meetings.

_ Design a survey form that can be easily tabulated.

_ Ask questions to provide information about what you

need to know: attitudes about each component of the

succession planning program.

_ Allow for anonymity and allow the respondents the

opportunity to provide additional comments.

Program

Progress

_ Examine individual movements

through the organization.

_ Design an instrument that will provide quantitative

data.

_ Include ‘‘pre’’ and ‘‘post’’ level of skill/knowledge in

design.

_ Tie evaluation items directly to program objectives.

Effective

Placements

_ Performance checklists.

_ Performance appraisals.

_ Critical incident analysis.

_ Self-appraisal.

_ Base measurement instrument on systematic analysis

of key positions.

_ Consider the use of a variety of persons to conduct

the evaluation.

Organizational

Results

_ Organizational analysis.

_ Speed of replacement.

_ Cost of replacements.

_ Cost of nonreplacements.

_ Turnover.

_ Involve all necessary levels of the organization.

_ Gain commitment to allow access to organization indices

and records.

_ Use organization business plans and mission statements

to compare organizational needs and program

results.

Exhibit 13-3. A Worksheet for Identifying Appropriate Ways to Evaluate

Succession Planning and Management in an Organization

Directions: Use this worksheet to help you identify appropriate ways to evaluate the

SP&M program in your organization.

In column 1 below, indicate the various stakeholder groups (such as top managers,

key position incumbents, line managers, and the SP&M coordinator) who will be

primarily interested in evaluation results on SP&M in your organization. Then, in

column 2, indicate what levels of evaluation—customer satisfaction, program progress,

effective placements, and organizational results—will probably be of prime

interest to each stakeholder group. Then, in column 3, indicate how evaluation of

SP&M may be carried out in your organization.

Column 1 Column 2 Column 3

Stakeholder Groups What Levels of Eval- How Should Evaluafor

Evaluation uation Will Probably tion of the SP&M

Be of Prime Interest Program Be Carried

to Each Group? Out in Your Organization?

Exhibit 13-4. A Sample ‘‘Incident Report’’ for Succession Planning

and Management

Directions: The purpose of this ‘‘incident report’’ is to track successor/replacement

experiences in your organization.

Answer the questions appearing in the spaces below. Be as truthful as possible

because the collective results of many incident reports will be used to identify program

improvement initiatives for the succession planning program.

Fill out this report for each position filled from within. (This report should be completed

in addition to any personnel requisitions/justification forms that you are to

complete.) Submit the completed form to (name) at (organization address) within 3

weeks after filling the vacancy.

Name of Departing Employee Job Title

Department Time in Position

Reason for Leaving (if known)

Name of Replacing Employee Job Title

Department Work Unit/Team

Time in Position Today’s Date

1. Describe how this position is being replaced (internally/externally).

2. Was there an identifiable ‘‘successor’’ who had been prepared to assume this

position previously? If so, briefly explain who and how the individual was being

prepared; if not, briefly explain reasons for not preparing a successor.

3. Who was selected for the position, and why was he/she selected?

4. If an individual other than an identifiable successor was chosen for the position,

explain why.

Approval

Management Employee Title

(Signature)