At 3:35 on a muggy June afternoon in 1975, Eastern Airlines Flight 66, inbound from New Orleans, joined the arrival queue for New York’s John F. Kennedy Airport. A thunderstorm was brewing, and several airplanes ahead of Eastern 66 warned of dangerous winds on their approach. The crew of Eastern 66 heard the warnings but decided to press on. At 4:05 p.m., half a mile short of JFK’s Runway 22L threshold, Eastern 66 struck an approach-light tower and slammed into the ground, killing 113 people. It was, at the time, the deadliest single-airplane accident in aviation history.

It was also one of the hardest to explain. A modern jetliner should have had enough power to counteract even the strongest wind gusts expected under the conditions; why couldn’t Eastern 66 escape? Government investigators were unable to answer this question, but a University of Chicago meteorologist named Ted Fujita poured over the details of the crash and concluded that the plane had been felled by a previously unknown weather phenomenon he called a microburst, in which a compact column of cold air blasts out of the bottom of a thunderstorm cell, heading to the ground. Microbursts are energetic enough to be a serious threat to an airline and, confusingly for the pilot, cause the plane to at first gain altitude before being driven violently toward the ground.

Cockpit recorders in Eastern 66 revealed that this scenario is just what happened. But Fujita’s theory proved controversial, and it wasn’t broadly accepted until 1985, when more detailed data from another accident finally convinced the skeptics. Soon after that, the U.S. Federal Aviation Administration (FAA) launched a program aimed at eliminating microburst accidents.

The heart of the program was a training course for pilots on microburst avoidance techniques based directly on Fujita’s theory. It was spectacularly successful. In the decade prior to 1985, over 750 people died in microburst-related commercial accidents crashes in the U.S. After the FAA’s training program began, only one fatal commercial accident due to a microburst occurred in the U.S. (in 1994).

Think about that for just a second. We often boast about ROI in corporate training world, but imagine creating a training course that saves 75 human lives per year. Talk about a return on an investment!

Continuous Improvement

What’s more amazing is that microbursts are just one threat out of thousands that have been neutralized over time through targeted pilot training. This cycle of investigation and training has resulted in a continuous, spectacular improvement in commercial aviation safety. Since the beginning of commercial aviation in the 1930s, the rate of fatal accidents has steadily and almost exponentially declined from one per million passenger miles flown to approximately one per three or four billion passenger miles flown now.

Applying the Lessons from Aviation in Other Domains

Commercial pilot training is one of the most amazing success stories in the history of education. It makes sense to ask what lessons we might learn from it that we could apply to other domains. The good news is that the techniques used to achieve these results can be applied in almost any other industry. The key is a cycle of improvement in which:

1. An incident is investigated.
2. Analysis determines a root cause.
3. A response to eliminate the problem is defined.
4. Training is created to teach that response to practitioners.

My colleagues and I have developed a general process for implementing such a methodology that we call Critical Mistake Analysis (CMA). We have implemented CMA successfully in domains that range from clerking a department store counter, to quoting insurance policies, to drilling an oil well. The central idea of CMA is that the challenges that merit the most attention in learning a skill are the ones that cause the most impactful mistakes when the skill is performed in real life.

Critical Mistake Analysis

Task 1: Investigate

The first step in investigation is to compile a list of the mistakes that practitioners of a target skill make in real life by auditing and observing the skill being performed; interviewing and surveying practitioners and experts; and tapping into existing data sources, such as records of customer service complaints.

The next step is to look for patterns of similar issues. For example, in general (as opposed to commercial) aviation, we might notice patterns like these:

• A pilot tries to eke a few extra miles out of a tank of fuel and runs dry in mid-air.
• A pilot tries to traverse a gap in a line of thunderstorms and is trapped in a storm cell when the gap closes.
• A pilot “buzzes” his girlfriend’s house and crashes. (I say “his” because, interestingly, I’ve found no instance of a female pilot buzzing her boyfriend’s house.)
• A pilot trying to land on a low-visibility day continues to descend despite not seeing the runway and hits an obstacle or misses the runway.

The next step in the investigation is to determine the criticality of each mistake to decide which ones are most important to address in training. Criticality is an estimate of the positive impact we can expect from training on this mistake – in other words, the value of training on that mistake. In general,

Criticality = F * C * R

where:

• F is the frequency of the mistake.
• C is the cost of making the mistake.
• R is the remediability of the mistake, meaning the percentage of all instances that we expect could be eliminated by training.

The first two criteria should be somewhat self-explanatory; basically, we would like to eliminate the mistakes that have the greatest overall impact, which is given by F * C. Remediability (a word we made up) takes account of the fact that some mistakes are easier to fix than others. Buzzing your girlfriend’s house is a (strangely) frequent, and very costly, mistake, but it is a mistake that is caused by stupidity rather than ignorance. Training does a much better job of fixing ignorance than stupidity, so addressing “house buzzing” might not have a big impact.

To account for this reality, remediability is defined as the percentage of the occurrences of a given mistake we expect a priori to eliminate through training. Mistakes that are caused by lack of knowledge or skill are assigned high remediability scores, while those caused by poor judgment, process issues or perverse incentives are given low scores.

The final step in investigation is a Pareto analysis, in which we list the mistakes in order of criticality and identify a cutoff point at which the value of addressing the next-most critical mistake no longer justifies the effort and expense of doing so. It is called a Pareto analysis because generally, the criticality of a given set of mistakes follows a Pareto, or “80/20,” distribution, with a small set of mistakes accounting for most of the overall negative impact. That’s good news, because it implies that focused training addressing the handful of mistakes at the top of the chart will have a disproportionate positive impact.

Task 2: Analyze

The next step is a root cause analysis of each mistake on our list focusing, in particular, on four components:

• Situation: the situational factors that make the mistake likely
• Decision: the critical decision that leads to the mistake
• Misconception: the misconception that motivates the wrong choice
• Consequence: the outcome of making the mistake, and why it occurs

For example, for the microburst problem, the analysis might look something like this:

• Situation: flying “low and slow” near a thunderstorm, especially while approaching for landing
• Decision: failing to attempt an escape as soon as a potential microburst condition is detected
• Misconceptions: (1) The airplane can handle whatever wind gusts it is likely to encounter, and (2) the throttle should always be cut back if the airplane floats above the glideslope.
• Consequence: described above in detail

Task 3: Define the Response

Once we have analyzed a mistake, the task of defining the correct response falls to subject matter experts. Often, the response is already known, just not by practitioners. In some cases, the response is fairly obvious, given the problem, while in a few cases, the response requires some effort to define.

Task 4: Train

The most effective way to train learners to avoid a mistake is to construct a realistic learn-by-doing scenario in which learners confront the challenge that typically leads to the mistake and practice executing the correct response.

An effective challenge meets three main goals:

• Ensure that the learners who would make the mistake in real life will make it in training.
• Make the simulated consequence of the mistake as memorable as possible.
• Make the situation as authentic as possible to ensure that learning is transferred to the real world.

A challenge in CMA consists of five components:

• Backstory: a description of the situation in which we initially place the user; for example, “approaching for landing in thunderstorm conditions”
• Action sequence: for example, a sequence paralleling what Eastern 66 experienced
• Decision point: for example, the last point in our scenario at which a pilot could throttle up and expect to escape
• Playout of consequences: for example, a crash sequence that is as faithful as possible to what happened to airplanes like Eastern 66
• Coaching and feedback: Learners may need help to understand fully the mistake they made and how it led to the observed consequences. Tailored coaching enhances the learners’ ability to process and assimilate the lesson they learned.

Once we have sketched challenges in this way for all of our targeted mistakes, we have completed our critical mistake analysis.

The Future of CMA

We have successfully utilized CMA in a wide variety of domains. In almost every case, the biggest challenge is data acquisition, because corporate training initiatives are not generally designed to process and analyze performance data continuously from the field. But the rapid digital transformation of the industry suggests that this problem will be less and less of an obstacle as more and more performance data is collected automatically. For example, airplanes now continuously track performance data on both equipment and pilots. The era of “big data” is at hand, and one of the most exciting uses of that data will be to enable a continuous improvement training methodology that allows us to implement a critical mistake analysis approach in every domain.