Checklists and Monitoring in the Cockpit: Why Crucial Defenses Sometimes Fail

This seminal study by Dismukes and Berman (2010) from NASA looks at the role of checklists and monitoring as two essential defences against equipment failures and pilot errors. Problems with checklist use and pilots’ failures to monitor adequately have a long history in aviation accidents. A short summary of the key recommendations is given here. For those interested in further details and the full study check out this LINK.

Source: NASA Dismukes, R., & Berman, B. (2010). Checklists and monitoring in the cockpit: Why crucial defenses sometimes fail (No. ARC-E-DAA-TN1902).

The Study

This study was conducted to explore why checklists and monitoring sometimes fail to catch errors and equipment malfunctions as intended.

A LOSA-style observation of 60 normal operational flights from the cockpit jumpseat was carried out at three airlines to attempt to record every observable
deviation from procedure by the flight crews, even the most minor, including deviations that may have been necessitated by operational conditions.

Results

Eight hundred ninety-nine deviations were observed (194 in checklist use, 391 in monitoring, and 314 in primary procedures).

Checklist deviations clustered into six types:

• flow-check performed as read-do;
• responding without looking;
• checklist item omitted, performed incorrectly, or performed incompletely;
• poor timing of checklist initiation;
• checklist performed from memory;
• failure to initiate checklists.

The first two types accounted for nearly half of the checklist deviations observed.

Monitoring deviations

Monitoring deviations grouped in three clusters:

• late or omitted callouts,
• omitted verification
• not monitoring aircraft state or position

Over half of the monitoring deviations were late/omitted callouts. Much more serious were omitted callouts during 11 approaches that were unstabilized, eight of which remained unstabilized beyond the final gate.

Although this study focused mainly on checklist use and monitoring deviations, additional data on primary procedure deviations provide context and allowed analysis of how effective checklists and monitoring were at trapping primary procedure errors. The 15 types of primary procedure deviations were grouped into six areas:

1. coordination within the crew or with ATC
2. use of automation
3. approach stabilization;
4. path and airspeed control;
5. configuration of systems and flight controls
6. planning and execution

By far the most common deviations were failure to properly configure systems (62 instances), poor planning for contingencies (57 instances), poor coordination between the pilots (56 instances), and problematic use of the FMS (40 instances). Most of these deviations appeared to be inadvertent and can properly be described as errors.

Factors affecting deviations

In contrast with an NTSB study of accidents attributed to crew error, this study did not find that flights running late produced more deviations. However, consistent with previous studies, it did find that crews on their first flight together or on their first day of flying together made substantially more deviations. First officers and captains in their first year in aircraft type and seat position did not make more deviations than pilots with more than one years in type and position, however the three airlines where operations were observed hired only pilots with substantial experience; thus this result might not apply to smaller airlines that hire pilots with substantially less experience.

Error trapping

Only 18% of deviations—even those that were clearly errors—were trapped (caught and corrected) or even discussed, a disquieting finding. In comparison, Klinect et al. (1999) reported that 36% of errors observed in LOSA were trapped, and Thomas and Petrilli (2006) reported 63% were detected and actively managed in a flight simulation study. The lower trapping rates in this study probably reflect multiple factors, one of which is that actual line operations were observed, in which operational pressures and opportunities for error are not fully captured by simulations.

Countermeasures – recommendations

A set of countermeasures were developed as recommendations to substantially reduce pilots’ vulnerability to deviating from SOP:

Cockpit Procedures and Organization Policies

Recommendations:

1. Formalize monitoring and challenging requirements and procedures.
2. Minimize checklist items involving multiple components and specify responses for each component.
3. Evaluate error vulnerability of existing procedures and strengthen them.
4. Organizations should periodically review cockpit operating procedures to identify and relieve “hotspots” in which prospective memory and concurrent task demands are high and interruptions are frequent.
5. Organizations should systematically analyze the entire body of explicit and implicit messages given their pilot corps to balance competing goals.
6. Organizations should examine the role of organizational procedures in vulnerability to error in the cockpit (as well as errors in the cabin, dispatch center, and maintenance hangar).
7. Pilots should be trained on their inherent vulnerability to checklist and monitoring errors, and on procedural measures and practical techniques to counter it.
8. Reinforce the responsibility of monitoring pilots to challenge deviations.
9. Develop techniques to provide detailed feedback to pilots on checklist and monitoring performance.
10. Place greater emphasis on checklist use and monitoring in air carrier flight standards(line checking) programs.
11. Develop formal mentoring programs for new first officers.