The Difference is in the Design

Evaluation

Q:  Management expects us to evaluate the effectiveness of our training programs but is reluctant to allow time for a needs analysis. How can we do this without analyzing training needs?

A:  A formal needs analysis may not be needed. Instead, build an informal needs analysis into your evaluation strategy. (See Analysis).

Before conducting the program, sample a small segment of the target population and collect baseline data (number of errors, customer complaints, grievances, accidents) whenever possible. Look at existing records for hard data. A focus group is also a quick way to gather this data. For soft skills, administer a brief questionnaire at the beginning of the program. Be sure to sample the same group in the evaluation study. (For more ideas, see the next question.)

Q: I am trying to develop an evaluation instrument for levels 2 - 4 (learning, transfer and results).  Any recommendations?

A:   I assume you are looking for different instruments for each level of evaluation. My all-time favorite reference for evaluation methods is Handbook of Training Evaluation and Measurement Methods by Jack J. Phillips. Chapter 5 covers common evaluation instruments for all four levels (reaction, learning, behavior and results).

Q:  How can I measure the effectiveness of soft skills training programs, such as leadership, time management and communication skills?

A:  Use Success Stories and Supervisor Testimonials.

Success Stories are actual situations provided by program participants. Provide each participant with a "Success Story" form to complete and return 30 days after training. The form should allow participants to describe how the skills or knowledge used in the training program helped to improve their job performance. Ask them to assign a dollar value to the improvement, if possible. Compile the results and include in a quarterly report to management. Also, be sure to provide recognition to participants.

Supervisor Testimonials are statements provided by the participants' supervisors or managers. Conduct critical incident interviews with supervisors or managers of participants who have completed the training program. Ask them for a comparison of the participant’s performance before and after training. Document testimonials of performance improvement and include it in your quarterly report to management. (Be sure to keep participants’ names confidential.) Encourage supervisors and managers to reward participants for improvement.

Q:  Are there standard methods of evaluating trainers' performance in the classroom?

A: It's important that you define the criteria for quality and measure your trainers against these criteria. Define trainer performance expectations upfront (e.g., delivers information clearly, answers participants' questions completely, etc.) and measure their performance with a well-designed questionnaire immediately after the course.

While you are developing the questionnaire, think about what your customers expect from the trainers. How well have their expectations been met? Avoid evaluating trainers on their personalities rather than their performance.

Have your trainers been trained in the same delivery techniques? If not, make sure they all go through the same train-the-trainer program to ensure consistency in course delivery.

For train-the-trainer resources, visit the web site of the American Society for Training and Development (ASTD) at http.//www.astd.org.

Q: Our department uses a 5-point Likert scale on our training reaction questionnaires. Instructors are expected to achieve a 4.7 average. Is there another way to measure reaction that doesn't turn into a popularity contest?

A:  The approach you describe focuses the goal of evaluation on achieving a high rating rather than measuring participant satisfaction and obtaining specific, constructive feedback for improvement.  

I prefer to use criterion-referenced or performance-based items on the questionnaire. For example, some performance-based items might include:

Instead of a 5-point scale, I provide four levels of agreement (Strongly Agree, Agree, Disagree and Strongly Disagree) to eliminate a neutral response (Neither Agree or Disagree). This gets more meaningful results. Also, I include a space where participants can explain the reason for their choices. 

Evaluation results can be quantified by figuring the percentage of responses for each choice. This approach gives a clearer view of the distribution of responses because weighted averages may be skewed by extreme values, 

Need help evaluating your training programs? Send email to achoden@trainingbydesign.com .

Send mail to achoden@trainingbydesign.com with questions or comments about this web site.
Copyright © 2005 Training by Design
Last modified: July 01, 2005