The Five Levels of Evaluating Coaching

К оглавлению
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 
17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 
34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 
51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 
68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 
85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 
102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 
119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 
136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 
153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 
170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 
187 188 189 190 191 192 

During the course of the OptiCom story, we learned that coaching

objectives should include the application of coaching in the workplace.

We want our leaders to learn from the coaching and gain

insights into their behavior; however, we also want them to apply

these insights to their workplace. As Table 11.3 shows, learning and

application are but two of a total of five formal evaluation levels.

This leveling scheme has been in operation in training and development

for more than 30 years and is a natural fit for evaluating

coaching.3

Level 1 looks at the initial reaction of the coaching client to the

first two or three coaching sessions. Specific areas to look at include

the following:

_ How well the foundation for the coaching relationship was set

_ How much rapport has been established

_ How clear the intentions for the coaching initiative were

understood

_ How effectively the coach and client agreed on behavioral

objectives

_ How effective coaching is being delivered (e.g., via telephone

or in-person), session length, and frequency

Developing an Evaluation Strategy 189

3Kirkpatrick (1998); Phillips (1997).

_ How well the assessment data were explored and explained

_ The pacing of the initial sessions (e.g., too fast or too slow)

The purpose of gathering these data early is to take stock of how

the coaching relationship has gotten off the ground and, if there are

issues, to deal with them early. The last thing anyone wants is for

issues to be left unattended so they can fester and impede the

progress of coaching. If, for example, after three sessions the client

is still not clear about the behavioral objectives he or she must

address, then this issue must be dealt with quickly in order for the

coaching to have focus and direction. If the client still has lingering

questions about his or her personal assessment data, then the client

and the coach need to explore these questions and incorporate the

learnings into the coaching conversations.

Data that capture the initial reaction of people to coaching are

grouped and summarized from all coaching participants. These data

Table 11.3 The Five Levels of Evaluating Coaching Initiatives

Level Description

Level 1: Reaction This level looks at the initial reaction of the coaching

clients to the coaching they received. Reaction data are

typically captured after the first two or three sessions.

Level 2: Learning This level captures what the clients have learned from

their coaching sessions.

Level 3: Application This level evaluates how well clients have applied what

they have learned (and committed to) during the

coaching to their workplace. All evaluation strategies for

coaching initiatives should include assessments at this

level.

Level 4: Business This level documents the business impact from the

Impact coaching initiative. Impact areas include output, cost,

quality, and time.

Level 5: ROI This level tabulates all monetary benefits from the

coaching initiative, factors in the fully loaded cost of the

initiative, and calculates the return on investment.

are important for the sponsors and managers of the coaching initiative

to review. There may be issues common to the participants that

must be dealt with in order for the initiative to be successful. For

example, if after three sessions most clients have not developed

objectives for their coaching, then some additional support would

have to be provided to the clients. Or if many clients are struggling

with the available assessment data, then perhaps the appropriate data

were not adequately collected during the initial stages of the coaching

process.Additional multisource feedback data could be collected

or an entirely new assessment process could be implemented. The

point here is that it is better to hear about these issues after three sessions

rather than during a debriefing at the end of the six-month

coaching initiative. The more that is learned at the beginning about

how to make coaching more effective, the fewer the number of

lessons learned that will be recorded at the end of the coaching.

Level 2 explores what the clients are learning from their coaching

experiences. Every client will have a unique learning experience, but

themes may emerge and topic areas common to many clients may

become evident. Examples of learning areas include discovering new

ways to do the following:

_ Be a more effective leader

_ Gain insights into personal changes required to grow and

develop

_ Better understand how personal actions affect others

_ Work more effectively with peers to accomplish business goals

_ Improve communication skills

_ Improve the ability to collaborate with other leaders

_ Increase teamwork

_ Embrace a big picture view of the organization

_ Think more systemically about finding solutions to problems

Each of these learning areas can be a stepping stone to accomplishing

a business objective. For those who sponsor or manage a

coaching initiative, it is not so important which stepping stones are

being touched, as it is to know that at least some stepping stones are

being touched. If halfway through a six-month coaching initiative

little of significance has been learned by many of the clients, then

something is not working. If people are not making significant

strides in learning how to be more effective leaders, then it is unlikely

they will do anything differently. One cannot apply what one has

not learned.

Level 3 is where the rubber meets the road. Evaluation at this level

looks at how well people have applied to the workplace what they

have learned from their coaching experiences. Of course, not all of

what people learn from coaching can be immediately applied, but

coaching conversations conducted in business settings will at some

point turn to what the client will do differently. These changes in

behavior are the subject matter for the evaluation of application. For

example, in Chapter 3, Jane was able to accomplish her top priorities

by focusing her time and energy more effectively. The actions

she took were tangible expressions of what she learned about herself

from the coaching sessions. Here are some other examples of taking

actions based on coaching:

_ More clearly setting priorities for major initiatives

_ Making decisions based on a greater diversity of input

_ Using vision and passion to motivate others

_ Recasting roles and responsibilities to smooth out working

relationships

_ Expressing appreciation for the contributions of others to

increase retention

_ Articulating problem statements in a way that engages people

to take action

In each of these examples, a client did something differently as a

result of coaching. These behaviors can be observed and recorded,

which begs the question: What data are required to conduct an

evaluation of application?

The answer to this question lies in how best to tell a story about

the growth and development of the coaching client. Virtually all

clients will apply at least something they learned to the workplace.

Recording percentages and rates of application while important,

don’t come close to telling the whole story.What is of interest is how

coaching stimulated the client to take bold and successful actions.

Each of the four stories of clients in Section One of this book chronicle

successful actions that were taken. These stories, while qualitative

in nature, pack an emotional punch. Sharing these stories (with

the client’s permission of course) can have a powerful impact on initiative

sponsors and business leaders about the value of coaching. In

a successful coaching initiative, almost every client will have a compelling

story to tell.

The stories and experiences of coaching clients can be summarized

to show areas of application.

Level 4 examines the success stories of the coaching clients to

determine the potential impact on the business. Taking action is one

thing, but having this action translate to business results is quite

another. There are four types of business impact: output, quality,

cost, and time. Output measures include such areas as productivity

and sales revenue. Quality measures include such areas as increased

product reliability and reduced manufacturing defects. Cost measures

refer to reducing costs in operations; sales, general, and administrative

(SG&A) expenses; manufacturing; and other areas. Time

measures include reduced cycle time and product-to-market time.

A coaching client may affect one or more of these areas, based on

his or her work responsibilities. A leader in research and development

may take actions that reduce the product development cycle

(time), whereas a sales leader may work in a new way with sales

representatives that leads to increases in revenue.

The impact on the business can be both monetary and intangible.

In a sense, all benefits are intangible until we make the decision

to convert these benefits to monetary value. Making this decision is

not to be taken lightly. A considerable amount of effort may be

required to convert benefits to monetary value. There are two major

avenues for converting coaching benefits to monetary value: expert

estimation and performance indicator tracking. Keep in mind that

at this point we are developing the evaluation strategy, so all that is

required is to plan which benefits will likely be converted to

monetary value and how this conversion will take place. The first

approach, expert estimation, means that the coaching clients, and

perhaps others, will give their best estimate of how their actions

created monetary value to the business. So, for example, a leader

who, as a direct result of the coaching she received, reduced the

product development cycle time by 15 percent would estimate the

annual monetary value of this improvement to be $250,000. She

would then state her confidence in this estimate to be 50 percent.

The benefit would be discounted by this amount to result in a value

of $125,000. As an option, additional steps may be taken to validate

the monetary benefit. For example, the leader’s coworkers could be

interviewed to see if they would agree with the 15 percent cycle time

reduction.

The second approach, which may be combined with estimation,

is to track established performance indicators. An important aspect

of determining business results is to isolate the effects of coaching

from other potential influencing factors (which is dealt with in more

detail later in this chapter). Briefly, these effects are isolated by

looking at performance indicators according to a base period and

then a treatment period. The base period refers to performance

before coaching, and the treatment period refers to coaching. So, in

this product development example, the performance of the product

development team could be examined after coaching and compared

to the performance before coaching. Changes in performance could

be attributed to coaching, all other factors being equal.

At this point of developing the evaluation strategy, it is important

to think through how business results will be measured. In all likelihood,

both estimation and metrics tracking will be used. The

advantage of building these measurement activities into the evaluation

strategy is that plans can be made to obtain the required data.

Also, sponsors of coaching and other business leaders can understand

and buy into how business impact will be measured. Gaining

this buy-in early will save a lot of headaches later. In situations where

the required data are not being tracked, having this early warning

allows people to put the appropriate measurement tracking in place.

Coming to this realization at the end of the coaching initiative may

be too late to collect the required data.

Level 5 looks at ROI. A coaching initiative may have produced

monetary benefits, but did these benefits outweigh the cost of the

initiative? This evaluation level factors in the cost of the coaching

initiative and,when compared to the monetary benefits, the ROI can

be calculated. The formula for ROI is as follows:

ROI = ((Benefits - Cost) Π Cost) ¥ 100

The ROI formula can be contrasted with the benefits-cost ratio

(BCR) in that ROI deals with net benefits whereas the BCR deals

with total benefits. ROI is expressed as a percentage whereas the BCR

is expressed as a ratio:

BCR = Benefits Π Cost

So, for example, a coaching initiative that produced $400,000 in

value and cost $200,000 to execute would produce the following

calculations:

ROI = (($400,000 - $200,000) Π $200,000) ¥ 100 = 100%

BCR = $400,000 Π $200,000 = 2:1

It is important to include both the ROI and the BCR given that

people often confuse the two. Some people, for example, will report

a “3 to 1 return on investment.”Upon closer inspection it’s clear that

they did not follow the standard formula and use net benefits. There

is no quicker way to lose credibility than to appear confused about the

standard formulas for showing monetary value for the investment.

Structuring an Evaluation Strategy Based on the Five Levels

At first, structuring an evaluation strategy that addresses all five

levels may seem to be a daunting task. How do these five levels fit

Developing an Evaluation Strategy 195

into an evaluation strategy? It really boils down to two options:

evaluate at level 3 application or evaluate at level 5 ROI. A central

tenet of this book is that every coaching initiative should be evaluated

at either of these levels.Which level is best? Let’s contrast these

two strategies and look at the reasons for conducting a level 5

evaluation.

Table 11.4 contrasts level 3 (application) and level 5 (ROI) evaluations.

Although both kinds of evaluations collect data on intangible

benefits, these benefits are the primary stated benefits of a level

3 analysis, whereas with a level 5 analysis, intangibles supplement

monetary benefits. Monetary benefits are central to a level 5 analysis

and are not included in a level 3 analysis. As such, isolating the

effects of coaching is critical to a level 5 analysis and not required

for level 3. Likewise, initiative cost data are not required for a level

3 analysis, although they are required with a level 5 analysis. The

Table 11.4 Contrasting Level 3 Evaluation with Level 5 Evaluation

Differentiator Level 3 Evaluation: Application Level 5 Evaluation: ROI

Intangible benefits Intangibles included as Intangibles included as

primary benefit a secondary or

supplementary benefit

Monetary benefits Not included, although Included as a central

incidental cost savings may benefit

be cited

Isolating the Not needed, given that no Central to establishing

effects monetary benefits are cited monetary benefits

resulting directly from

coaching

Initiative cost Cost data are not collected Cost data are collected

for the purposes of the and, when compared to

evaluation benefits, the ROI is

calculated

The punchline How what was learned How what was applied in

of the story from the coaching was the workplace created

applied in the workplace monetary value for the

business

punchline of the level 3 story is how what was learned from the

coaching was applied in the workplace. For level 5, the punch line is

how what was applied in the workplace created monetary value for

the business.

Given that at least a level 3 evaluation is recommended for every

coaching initiative and is also conducted for every level 5 evaluation;

the real question becomes whether or not to evaluate at level 5. Let’s

examine the reasons to conduct a level 5 evaluation for coaching:

_ The business leader or initiative sponsor asks for it. This was the

case with OptiCom, and although Jacqui pushed back on the

request, she did construct an evaluation strategy for the full

deployment of coaching.

_ The coaching initiative has especially high visibility. If a lot is

riding on the outcome of the initiative, it is probably worth the

investment of time and money to ensure that the organization

gets the most out of the initiative.

_ The initiative (or pilot) is the first of many deployments of coaching

throughout the organization. Evaluating the first of many

deployments will enable the learnings from the evaluation to

be leveraged across subsequent deployments. For example, at

OptiCom, coaching will be deployed to 80 leaders. Still in the

offing is potential deployment to an additional 250 leaders,

who represent the next lower level of management. Leveraging

the learnings from the 80 to improve the deployment

and coaching experiences of the 250 represents an excellent

investment.

_ You, as a learning or change leader, need to learn as much as you

can about how coaching created value. A level 5 evaluation provides

a deeper understanding of the sources of value for coaching,

the areas in the organization that were impacted, and how

a monetary ROI was realized overall.

Any one of these four reasons is sufficient to launch a level 5 evaluation.

If none of these reasons is met, then a level 3 evaluation

Developing an Evaluation Strategy 197

may be best. Evaluating application provides insights into how the

coaching clients utilized their learning to create value for the organization.

This value, while intangible, may nonetheless represent a

substantial benefit to the business.

Conducting an evaluation is a more formal way of telling a story.

Every coaching client has a story to tell. Each level of evaluation represents,

in effect, chapters in these stories. It is important not to leave

any chapters out. So, when evaluating at level 3, consider beginning

the story with the initial reaction (level 1) of the client to the coaching,

the foundation that was set, and the rapport that was built. This

foundation supported the client in gaining insights into his or her

behavior and discovering ways to grow and develop (level 2). These

insights and discoveries led to taking bold new actions and directions

to impact the organization (level 3). Filling in these three chapters

tells a great story.When evaluating at level 5, the story continues

by showing how these actions and directions created value for the

business (level 4) in such a way that the coaching initiative more

than paid for itself (level 5).