Evaluation of a human factors analysis and classifi cation system as used by simulated mishap boards

Background: The reliability of the Department of Defense Human Factors Analysis and Classifi cation System (DOD-HFACS) has been examined when used by individuals working alone to classify the causes of summary, or partial, information about a mishap. However, following an actual mishap a team of investigators would work together to gather and analyze a large amount of information before identifying the causal factors and coding them with DOD-HFACS. Method: There were 204 military Aviation Safety Officer students who were divided into 30 groups. Each group was provided with evidence collected from one of two military aviation mishaps. DOD-HFACS was used to classify the mishap causal factors. Results: Averaged across the two mishaps, acceptable levels of reliability were only achieved for 56.9% of nanocodes. There were high levels of agreement regarding the factors that did not contribute to the incident (a mean agreement of 50% or greater between groups for 91.0% of unselected nanocodes); the level of agreement on the factors that did cause the incident as classifi ed using DOD-HFACS were low (a mean agreement of 50% or greater between the groups for 14.6% of selected nanocodes). Discussion: Despite using teams to carry out the classifi cation, the fi ndings from this study are consistent with other studies of DOD-HFACS reliability with individuals. It is suggested that in addition to simplifying DOD-HFACS itself, consideration should be given to involving a human factors/organizational psychologist in mishap investigations to ensure the human factors issues are identifi ed and classifi ed in a consistent and reliable manner.