Leader.co.za - Management, Training and Career Advice for Business Leaders





COMENSA

Telephone: 010 443 5081
Fax: 086 669 0131
E-mail: Send us an e-mail
Web-site: Visit our web-site
Postal Address: PostNet Suite 532, Private Bag X4, SUNVALLEY, 7985
26 MARCH 2008
Assessing the effectiveness of coaching

by Mongezi Makhalima: CEO of The Entrepreneurship Academy of Southern Africa and Executive Coach to some of the top leaders in the country and on the continent.

A lot has been said and even more written about how organisations can measure the effectiveness of coaching.

It has become even more important for the organisations spending money on coaching interventions to know and assess where their investment goes. What this implies is that coaches are now expected to balance the person-centred approaches to measuring coaching effectiveness with those that can also show value of these coaching interventions to individuals in organisations.

It is generally accepted that at the primary level, the effectiveness of coaching is in the “lived experience” of the client. It however still does not satisfy the need of the Head of HR in business, who needs to justify the investment in coaching to the CEO.

The Kirkpatrick Model

While there has been a debate in general about what coaching is and is not, there seems to be a growing consensus that it is a process that enables learning & development to occur (Parsloe, 1999; Downey, 1999; Grant, 2000; Clutterbuck 2003) and not some mystical process with immeasurable outcomes.

I would like to propose perhaps another way of looking at measurement of coaching outcomes for business. This paradigm will perhaps provide a different approach to the concept of the “value” of coaching.
 
If it is accepted that coaching is a learning intervention, it perhaps makes sense that coaching effectiveness can then also be subjected to the same methodologies used for assessing the effectiveness of learning/training.

In 1975 Donald Kirkpatrick introduced a four-level model of evaluation that has become a classic in the industry.




Figure 1: The Kirkpatrick Model


The model itself is self-explanatory, which is probably the reason for the wide acceptance thereof.
 
Level 1 measures the learner’s reactions to the training (sometimes referred to as happiness ratings). Level 2 measures whether or not there was a change in the skills, knowledge or attributes of the learners. Level 3 measures the change in behaviour that may have occurred as a result of the training while Level 4 measures the value that the training programme have resulted in for the organisation.

If one replaces a training programme with a coaching intervention, the model would look something like figure 2.




Figure 2 The Kirkpatrick Model adapted to coaching


Looking at the model now, one gets the sense of the kinds of questions that need to be answered by the organisation at every level in order to assess the success of coaching outcomes.

The ROI and the Kirkpatrick Model

The Manchester study (Manchester Review, 2001) already started suggesting the use of this model in measuring the effectiveness of executive coaching. In the study they also give good examples of the kind of questions that were asked at each level (even though this was more at research level).

In 1997, Jack Phillips, the chairperson of the ROI Institute, introduced the ROI (Return on Investment) measure as a fifth level to the Kirkpatrick model. Unfortunately, it is this level which received more attention in the Manchester study.

While the exposition and treatment of this measure was useful, it served to divert the attention from what I believe are other important elements in the measurement schema.

I suggest that the original four levels in the schema are sufficient for the purposes of providing comfort to modern businesses regarding the impact of coaching.
 
Since business results are measured in more than just the ROI, I posit that the ROI measurement, if required, can be measured as part of level 4 measurements (and not as a separate level of measurement in the Kirkpatrick Model as suggested by Phillips).

I argue that a ROI measurement is a global business performance measurement, which cannot be attributed to just one element in the organisation - the improvement in business performance may not necessarily be as a result of a single intervention, but may be coming from perhaps improved sales processes or better products. To therefore focus on ROI may not necessarily be the answer after all.

In fact, if the organisation finds it really important to measure the global effect of coaching specifically, it makes a lot more sense to measure it within the “learning & growth” perspective of the organisation’s balanced scorecard. In this way the organisation is able to attribute the investment in learning closer to “source”.

The calculation can look like the following simplified equation

Cost of coaching initiatives + cost of other learning interventions = Total cost of learning.

This cost can then be included as part of the Net profit margin, which is the one part of the ROI calculation.

The value of dialogue over numbers

I prefer to open a dialogue with my clients that lead them to an understanding that’s deeper than the numerical dynamics of the intervention. I find that a lot of leaders in organisations have accepted that the impact of learning cannot be assessed only in a reductionist manner. They are generally aware of the systemic interactions of elements in their businesses.


Figure 3 Level four evaluation attempts to assess training in terms of business results. In this case, sales transactions improved steadily after training for sales staff occurred in April 1997. From Winfrey, E.C. (1999). Kirkpatrick's Four Levels of Evaluation In B. Hoffman (Ed.), Encyclopaedia of Educational Technology. http://coe.sdsu.edu/eet/Articles/k4levels/start.htm 


I have found that taking a balanced scorecard approach; especially what Kaplan & Norton (1996, 255) refer to as Echo-Engineering (figure 4), helps to highlight the systemic interactions of different perspectives. Once shared understanding has been achieved, the scene has been set to open dialogue on more detailed measures using the Kirkpatrick model.


Figure 4 Echo Engineering - Linking measures from the four perspectives


To paraphrase Flaherty (1999), Coaching is not a matter of accounting - adding up positives and negatives about the programme… applying a formula that extends the calculation into the future, and then allowing the dynamics of the formula to determine whether coaching was successful or not.
 
I believe that the success of any coaching intervention starts with clarifying the purpose for the coaching intervention, followed by an honest dialogue around expectations and what they would look like to both the organisation and the coachee.

While facilitating this conversation, it is important to keep in mind that coaching is not about balancing the numbers; it is about developing individual capabilities for exponential collective impact. Like trust, respect and loyalty, it has no formula.

Acknowledgements

  1. Flaherty, J. (1999). Coaching-Evoking excellence in others. Massachusetts: Butterworth-Heinemann
  2. Gitman, L.J (1991): Principles of managerial finance. 6th ed. New York: Harper-Collins.
  3. Kaplan, R.S & Norton, D.P (1996). The Balanced Scorecard- Translating strategy into action. Boston: Harvard Business School press.
  4. Kirkpatrick, D.L & Kirkpatrick, J.D (2005): Transferring Learning to Behavior: Using the Four Levels to Improve Performance. San Francisco: Barrett-Kohler.
  5. McGovern, J. (2001). ‘Maximizing the Impact of Executive Coaching: Behavioral Change, Organizational Outcomes and Return on Investment’. The Manchester Review. Vol. 6(1). No.1
  6. Phillips, J. & Stone, R (2002). How to Measure Training Results- A Practical Guide to Tracking the Six Key Indicators. New York: McGraw-Hill
  7. SABA white paper (2001): Analytics-Understanding the Economics of learning. Saba Software Inc. California.
  8. Winfrey, E.C. (1999). ‘Kirkpatrick's Four Levels of Evaluation’ In B. Hoffman (Ed.), Encyclopaedia of Educational Technology. http://coe.sdsu.edu/eet/Articles/k4levels/start.htm
Source:

COMENSA
Home for outstanding South African coaches and mentors where they can meet, network and learn more. Visit our InfoCentre or website.

Share: Facebook
Facebook Twitter
Twitter LinkedIn
LinkedIn Email
Email
Share
Other Print
Print Newsletter
Newsletter



About Us


Features


News


Tweets


Events


Contacts