Measuring and Maximizing Training Impact

Bridging the Gap between Training and Business Result

P. Leone

• Publisher: Palgrave Macmillan
• Year: 2014
• Print ISBN:
  9781137414793
• eBook ISBN:
  9781137410481
• Topic: Business
• Subtopic: Business Strategy
• Index: Business

This book shows trainers how to create building blocks, construct the right linkages, and measure the impact of training programs from the first step (Level 1 – reaction) to the final destination (Level 5 – ROI). Including a new ground-breaking Level 6 exploring training sustainability, this is a must-read for HR professionals.

CHAPTER 1

Did They Like It?

How to Collect, Analyze, and Report Level 1 Data

Level 1 data, which captures employees’ reaction and satisfaction with a training experience, is collected by at least 70 percent of all organizations that offer employee training programs and curricula. It provides a way to measure engagement and get an overall impression of the employee experience. While fairly simple and unscientific, it is a great way to get an initial read on the face validity and relevance of the training content. It also serves as an opportunity to gather initial feedback on what’s working or not working in terms of design, duration, and facilitator effectiveness. These initial results will allow you to tweak your training design, approach, and/or content, and perhaps inform your trainers how to keep their audiences more engaged.

While there are endless possible questions you can include depending on what your initial goals are, there are only five core questions that you really need to ask:

1. Are you satisfied with the content (engaging materials, exercises, discussions, etc.)?

2. Are you satisfied with the delivery (facilitator or other delivery mode)?

3. What are the best and worst parts? (open-ended)

4. Overall, are you satisfied with the experience?

5. Would you recommend it to others?

With some version of these short and sweet questions you will be able to gather what you need to develop your full Level 1 report. Remember, a lot of Level 1 is good information to have, but why waste your time, and more importantly your survey-taker’s time, when the results are not necessarily predictive of higher levels of impact. If you are indeed reporting up to Levels 3, 4, 5, and especially 6, these early levels become far less important and will fast be forgotten as you present your more comprehensive analyses. So keep it short. Here’s a quick summary of why I included these five items:

1. Are you satisfied with the content (engaging materials, exercises, discussions, etc.)?

If the materials are dull, outdated, or incomplete, the best magician or wittiest comedian couldn’t keep the audience engaged and awake for more than an hour no less a whole day. That’s why it’s important to ask this question, but the big mistake many organizations make here is trying to hit on every possible thing that could be right or wrong with the training. If you try to cover every nuance of the training, you’ll end up with a list of about 25 questions ranging from “Did you get all the materials?” to “How did you like Module 5 Section 8?” This is not going to endear you to your survey-takers and you’re certainly not going to create a spirit of cooperation. Just keep it simple. Ask them if they’re satisfied with the training content and give them the option of “Yes” or “No.” If it’s a “No,” give them an opportunity to vent about the one big problem they had with it. If you ask it 25 different ways and want 25 different ratings, they’ll be checking off boxes just to get the heck out of there. Then you’ll get no valuable data back. There’s no use wasting their time and yours (remember you have to analyze all this stuff) if you could nail it all down with one question.

2. Are you satisfied with the way the training was delivered (facilitator, online experience, etc.)?

Here’s your opportunity to fire that obnoxious orator who shows up late, falls asleep during breakout discussions, and clearly has no idea what he or she is talking about. Granted, these cases are extremely rare and facilitator ratings almost never veer away from their overwhelmingly positively skew, but we really still need to ask the question. While the question will never add up to any informative robust data, it’s still important to detect that 1 in 50 cases where the instructor is truly murdering the chances of participants gaining anything valuable from the experience. This is why most vendors and training organizations insist on keeping it in their Level 1. If you are delivering the training online or using other media, the aggregate results become a little more actionable. Here, you can uncover things like technology issues and challenges that can have a widespread effect on your learners. Even if you’re using multiple modes of delivery within one training program, you can still cover it all with one question. Whatever delivery modes you are using, get the “Yes” or “No” and let them add their specific color commentary if they feel the need to do so. If they have the facilitator from hell, you’ll hear about it. If they couldn’t log on to your system, you’ll hear about it. If they don’t call it out here, it probably wasn’t that important in the first place.

3. What are the best and worst parts? (open-ended)

If ever they had an opportunity to highlight something important, this is it. This is where you’ll get some good feedback on the drivers and detractors of participant engagement. I’ve seen many programs evolve and improve based on these responses. For instance, in one training program a vast majority of participants said they loved the table discussions with their peers and benefited from the networking. Analyzing and really hearing that feedback, the vendor went off and worked several more table discussions into the training design for the following year. Similarly, one web-based training program received a huge number of negative responses pointing to the scenarios included in simulation training. The vendor made sure to rewrite the scripts and enhance all the scenarios for the next rollout. If you’re open to and looking for opportunities to upgrade your training, pay close attention to these comments.

4. Overall, are you satisfied with the experience?

This is purely “Yes” or “No” without any additional comments. I keep this question for one reason and one reason only—to summarize my Level 1 quantitative data. That is, if someone wants a quick snapshot of how all participants reacted to the training, this data point is what I use. As you’ll see (hopefully) when you’re reporting later on, your stakeholders are going to want a quick slide or a few brief bullets on Level 1 and then it’s onto the more comprehensive and informative parts of your impact story. This is why you want a broad enough question to cover all levels of satisfaction. You don’t want to be forced to look across all questions and comments and pull together some quantitative average. The response distribution to this question will give you that one headline you need for future reporting (e.g., “Overall, 92 percent of participants were satisfied with the training experience”).

5. Would you recommend it to others?

This one I like because it provides some variability. In my experience, participants tend to be a little less generous with this question than with many others I’ve seen on Level 1. For instance, participants might give rave reviews to facilitators and sing praises of how great the content was, but when it comes to recommending the training to a coworker, they suddenly become a little more critical. Saying you’d recommend a training program to your peers says a lot more about the power of that experience than simply saying you are satisfied. Because of this greater variability, it’s a good measure to track if you are continuously making changes and upgrades to your training program.

Scaling All Your Survey Questions

When it comes to developing a great, concise evaluation story, something that is almost as critical as the questions themselves is the scale or response options you provide your survey-takers. I will return to this important topic throughout the book because a little strategy and forethought about scaling will really change the power and impact of your reporting. What seems like small-scale nuances and petty “word-smithing” now can vastly improve the flow and impact of your results presentation later on.

For now, here are my three rules of thumb on building your actual scales:

1. Don’t use a neutral option in your scale. One of the most mystifying things to me is the amount of scales found across surveys, organizations, and even disciplines that include the infamous “no opinion” or “neither agree nor disagree” right smack in the middle of the scale. For the sake of your data analysis, make your raters commit one way or another. How can you go through hours or days of a training program and not have an opinion? Remember, you’re not asking for an ethereal truth scribed in stone—you just want an opinion, perception, or impression, and everyone has one. You know the saying about opinions right?

2. Whenever you can, try to keep your scaling to discrete answers (e.g., Yes or No) as opposed to the typical Likert-type scaling (e.g., strongly agree, agree, somewhat agree, etc.). For instance, if you ask the question, “Would you recommend this training?” and use the Yes/No options, you can report confidently and concisely:

• 90 percent of participants would recommend this training

Alternatively, if you ask the same question using the “strongly agree” to “strongly disagree” continuum, you’ll be stuck reporting a longer, blurrier response distribution like.

• 10 percent strongly agreed, 22 percent agreed, and 32 percent somewhat agreed with the following statement—“I would recommend this training.”

Now which bullet would you rather include in your report?

3. Be specific with your response options. For instance, if you’re asking participants to “[d]escribe the frequency of meetings you have with your leader to discuss your development plan,” try not to give them options like “Very frequently” and “Less frequently.” Just give them clear and specific response options like “Once a week,” ”Once a month,” “ Twice a year,” and so on. This leaves far less room for subjective interpretation of the word “frequently” and relies solely on the number of actual meetings.

Table 1.1 Example of a Level 1 survey

See table 1.1 for an example of a full Level 1 su...

Table of contents

• Cover
• Title
• Introduction  Bridging the Gap between Training and Business Results
• Chapter 1  Did They Like It?
• Chapter 2  Did They Learn Anything?
• Chapter 3  Are They Doing Anything Different?
• Chapter 4  Did It Impact the Business?
• Chapter 5  Was It Worth It?
• Chapter 6  How Do We Maximize Impact?
• Summary
• Conclusion
• Case Study 1  Measuring the Impact of Leadership Training
• Case Study 2  Maximizing the Impact of Leadership Training through Different Delivery Modes
• Index