Application objectives help define the behaviors that learners should demonstrate on the job after attending a learning event. However, acquiring data that supports this behavior change is often met with many barriers.

Here are a few ways to collect data and understand if behaviors have changed as intended.

What Is Level 3 Data? 

Level 3 data in the Kirkpatrick Model is data that shows behavior change and improved on-the-job performance. This data is critical to understand if learners can apply new skills, behaviors and knowledge to their job. Level 3 application data comes in many forms. It could be the number of times a manager has documented a development conversation, or it could be the percentage of time a new process is completed successfully. While it’s nice to have plenty of options, it also makes evaluating performance slightly more difficult.

Why Is It So Hard to Get?

The range of possibilities for Level 3 application data is vast. That data is usually available, but it is often spread out across the organization. When that data is so spread out, it’s going to require cooperation from multiple parts of your organization. Unfortunately, it’s not always easy to acquire. Sometimes there are data integrity issues in which the data isn’t as reliable. Other times, you’re looking for data that doesn’t even exist without a manual process. Lack of access or availability of data is a significant reason why learning and development (L&D) teams struggle to demonstrate impact.

What to Do About It?

While obtaining performance data seems like it’s the best avenue to understanding the impact of your training, it might slow you down. The amount of time spent gaining access to and cleaning the data so you can it use may be costly.

Here are a few ways to obtain evidence of behavior change while limiting time spent chasing down data:

Post-event and Follow-up Surveys

Using surveys to collect data is common. Likely, surveys are already a part of your delivery.  However, most surveys don’t go beyond gathering feedback on the course or the delivery itself.  Surveys can be a powerful tool if you ask the right questions. Using follow-up surveys and asking learners how much of the content they are using, and if it applied to their job, is a way to receive feedback. Learning managers are also helpful. Survey them to find out if course content is being applied on the job. Wait at least 60 days before sending out follow-up surveys. Give learners time to apply what they’ve learned on the job in a way that they see sustainable improvement.

To take surveys to the next level, consider asking questions about how much of the content will be applied to the job. It might also be helpful to ask if the learners feel they will have an immediate opportunity to apply what they learned. All these questions are leading indicators of learning transfer and will help determine if the content will be applied.

Be prepared for push-back on using self-reported data. Be prepared to discuss the pros and cons of self-reported data instead of assuming it’s unreliable. Shreya Sarkar Barney, Ph.D., president of Human Capital Growth completed a study on self-reported data and compared it to manager interviews, along with statistical performance data. The research shows that there was a negligible difference between the three methods.

Observations

Another method of understanding behavior change is live observations with learners after training. During these observations, you’re watching for specific behaviors that learners may or may not exhibit. Collect data from various learners on how successful they are, then aggregate the data from your observations. Ultimately, you want to understand the percentage of learners that are showing behavior change. There might be various behaviors you’re looking for, so note them separately. Remember, you’re looking for sustained behavior change. Space the observations out to determine if learners not only gained knowledge in training but can still apply it a few days or weeks later.

If in an environment where people are recorded while doing their jobs, such as call centers, consider using recordings for observation as well. The process is the same as live observations, except you’re using recordings.

Performance Data

If you must use performance data, prioritize data that already exists. During needs analysis, you may have determined several application objectives related to the course. Some of them will likely be data that is tracked and exists in your organization, and some might not. If you need to use real data collected from various systems, prioritize what is available. This will help avoid manual processes to collect and analyze data while giving a greater chance of obtaining data that you can use.

If you take this route, chase down this data early on. Don’t wait until the course is delivered, or a few months later, when you’re expected to report out on it. If you want to use real performance data, it’s going to take time to get access it, collect it, and then put it in a format in which you can use it with training data. Also, make sure to set the right expectations with sponsors. Let them know when they can expect to see results. If you’re using performance monitoring, it’s going to take extra time if you don’t already have access to the performance data.

Conclusion

In a world where showing results is more important than ever, you must have a clear picture of how to collect important data for your projects. The methods discussed here are all effective ways of collecting valid data to understand on-the-job behavior change.

As with most other business decisions, each comes with its own pros and cons. Take time to understand what behavior changes your business leaders expect to see and what metrics are impacted by those changes. Align on collecting data using one of these three methods, so you can go beyond smile sheets and start delivering results that executives want to see.