Several months ago, I wrote an article called “Measuring Training Impact for the Fourth Industrial Revolution.” The main point was: The only companies that will truly thrive in the future will be those that serve their customers in the simplest and fastest way, but somehow make it all feel more personal. This includes companies in the corporate training industry. Training must be more personalized and targeted to provide greater benefits, but also quicker and easier to keep costs down.
This case study explores what many consider a “futuristic” approach to training — the virtual reality (VR) experience.
A Real-world Case Study
VR as a training delivery option has been around for a while, but there have been few studies done to prove its value. How does it stand up against traditional delivery modes like classroom, online and blended learning? More specifically, what is its effectiveness when it comes to our six levels of training impact? Do participants like it better? Do they learn more? Is it better at improving behaviors on the job? Does it have more impact on the business? Does it carry a higher return on investment (ROI)? Are there factors back on the job that maximize that ROI? Let’s take a look.
The organization in this study was a multi-billion dollar Fortune 100 company, and the methodology compared two cohorts of 251 salespeople — one cohort was trained solely in a traditional classroom experience, the other cohort was trained in a half-classroom, half-VR environment. In essence, the entire group came together for core instruction during their first day of training and then broke out into these disparate groups for the second day.
As an evaluator, this provided a unique opportunity to isolate the impact of VR by controlling for so many other extraneous variables that typically confound these types of comparison studies. One group trained using VR, the other using traditional classroom. Would there be any differences in learning? And more important for the stakeholders: Would there be any difference over the next several months in the application and effectiveness of this training back on the job?
Summary of Results
Level 1 Reaction: The participants preferred the VR experience over the classroom, with an overall rating of 4.9 vs 4.3 out of 5.0. Even though both were engaging, and there had been some hands-on simulations in the classroom to make participants feel like they were on the job, it didn’t compare to the captivating and immersive experience of VR.
Level 2 Learning: We used a post-test only method where we asked participants to score where their knowledge and confidence was before — and after — the training. Participants reported a 19.8% higher level of learning from the VR experience compared to learning in classroom alone.
Level 3 Behavior Change: We measured improvement in six key behaviors two months after the training. Through a series of self and manager observations, we saw a “significant” to “exceptional” improvement in 67% of the virtual group, while only 34% of the classroom group showed the same high level of improvement.
Level 4 Business Impact: We knew we had to carefully “isolate” the impact of each of the delivery modes, so we compared both groups’ metrics over the three months post training and compared this key performance indicator (KPI) improvement to another group of employees who were not trained. This way, we could compare the respective increases in business performance to a true control group and attribute any improvement more validly to the training. Here, both groups outperformed the untrained group, and the VR group significantly outperformed the classroom. While there were three KPIs considered in the analysis, the main metric used and the easiest to monetize was the “new customer acquired.” The benefit per participant was calculated at $1,260 above that of the classroom alone. Again, both methods had a significant impact on the business, but the VR experience was clearly more effective.
Level 5 ROI: The calculation was straightforward: How did the additional benefit created by the VR compare to any additional cost for the VR? In total, the enhanced VR experience ended up costing each participant an extra $560, but when compared to the extra benefit per participant of $1,260, the ROI on the additional investment came in at more than 128%.
Level 6 Transfer Climate: We found that immediate manager support after the training and any applied insights from the VR experience (which can be captured and recorded for later coaching) led to a 48% increase in ROI. That is, if participants were able to capitalize and act on the data from the VR experience, they were able to reap even greater benefits.
Conclusion
There isn’t much argument around the fact that the closer it simulates real life, on-the-job experiences, the more effective training will be. We all aspire to make training as realistic as possible. The question is: How much additional business benefit and ROI can we achieve by edging ever closer to the immersive experience? While this is only one case study, that only sought to compare in-classroom training and VR training, it was an impressive glimpse into the promise of VR as a powerful training option. We already knew it can help you dive off the edge of a skyscraper, but now it seems like it may also make you a better employee.
