Evaluation of a human factors analysis and classification system used by trained raters

Background: The U.S. Department of Defense (DoD) has used the DoD Human Factors Analysis and Classification System (DoD-HFACS) to help identify and classify human factors that may have caused or contributed to aircraft mishaps since 2005. Method : In this study 22 military officers used DoD-HFACS to classify information obtained from an interview with an individual who had been involved in an aviation incident in which the potential for serious injury had been high. Results: It was found that although the overall interrater reliability was generally acceptable (as reflected by a mean Fleiss' kappa of 0.75) and there were high levels of agreement regarding the factors that did not contribute to the incident (there was agreement of 50% or greater between raters for 84.4% of unselected nanocodes); the level of agreement on the factors that did cause the incident as classified using DoD-HFACS were lower than desirable (agreement of 50% or greater between raters that a particular nanocode was causalwas found only for a mean of 22.5% of selected nanocodes). Discussion: The findings from this study are consistent with the small number of other studies reporting an evaluation of the reliability of DoD-HFACS. It is recommended that organizations must evaluate the reliability and validity of mishap coding systems, as applied by the proposed end-users, prior to the widespread adoption of a system. It is only through the accurate identification of mishap causalfactors that informed decisions can be made to prevent future mishaps.