Note: References to STEP have been superseded since 1989 by MES (Multilinear Events Sequencing) methods.
Computer software is now available to implement the ideas advanced in this paper.
Please also refer to 2005 supplement.
|A METHODOLOGICAL APPROACH TO THE SEARCH FOR INDIRECT (HUMAN) EVENTS RELATED TO MISHAPS
A presentation to
FOURTH INTERNATIONAL SYMPOSIUM ON AVIATION PSYCHOLOGY
The OSU Aviation Psychology Laboratory
The Association of Aviation Psychologists
April 26-30, 1987
Dr. Kingsley H. Hendrick|
Transportation Safety Institute
Oklahoma City, Oklahoma
|Ludwig Benner, Jr.|
Events Analysis, Inc.
AOPA Air Safety
This paper proposes the use of a disciplining methodology based on multilinear events sequencing to document and display human decision events associated with accidents. That methodology, Sequentially Timed Events Plotting (STEP), has been used extensively in the investigation and analysis of aviation and other of accidents. Its application to the understanding and improvement of decision events in aviation is discussed here.
Reported accident data, especially those reporting human factors information, tend to be abstract, to reflect judgments and disciplinary jargon, and to lack specificity. These attributes of accident data have impeded significant advancements in the understanding, prediction and control of safety-related problems in aeronautical and other fields of human endeavor.
ABSTRACT VERSUS CONCRETE TERMINOLOGY IN DOCUMENTATION OF ACCIDENT DATA
The ladder of abstraction shown below provides a useful way of visualizing the relationship between 'abstract and "concrete" words.
The concrete words at the bottom of the ladder substitute for pointing to specific, unique and observable phenomenon. Words found on higher rungs are called 'abstract" since they abstract only the "meanings-in-common" from all the words in their supporting ladders. The term "human factors" is relatively high on the ladder of abstraction. So, too, is "human error." These illustrate why an attempt to develop a ladder of abstraction, using human factor terms was frustrated. The ladder was essentially without a base, without concrete. "point-to-able" events.
Relevance to Understanding Accidents
The relevance of the ladder of abstraction to accident investigation and subsequent research is that the higher the level of abstraction, the lesser the specificity of the accident descriptions and the greater the difficulty in communicating problems and describing solutions to decision-makers. Without a concrete description of the accident process, one from which the accident scenario can be visualized, the probability of reaching agreement about what should be done to produce better human decisions, especially in the cockpit of airplanes, is significantly reduced. Without agreement among researchers and "investigating strangers" on the referents of words and terms, each may be looking at a different "elephant," as in committee folklore.
PROBLEMS ENCOUNTERED BY ONE RESEARCH EFFORT
Overview of the Research
At a meeting sponsored by the NTSB in 1984, one attendee made an impassioned plea for human factors information that could be used to make decisions, for data specific enough to be used by a pilot walking out to the flight line in making the decision to fly or turn around. As a result of this plea. in 1985 a research team undertook a search for a new approach to the Understanding and teaching of pilot decision making. The purpose of the research was to develop the teaching materials that would respond to that plea. A recent paper by Russell Lawton of AOPA describes the research. The research was based, in part, on the DECIDE decision making model developed for the hazardous materials transportation field. Certain modifications to the original DECIDE model were made, using work done during refinement of the STEP methodology. The research resulted in more successful simulator flight outcomes by subjects who had been instructed in the new approach than by others.
Development of the Decision Making Model
Relation of DECIDE to other Models. Although the DECIDE model closely paralleled Rockwell's aviation-based model, DECIDE' s focus on "change" as the trigger for a "decision set" represented a major difference. More importantly for training, DECIDE viewed decision making as a continuous, adaptive process made up of serial and parallel decision sets. This process perception parallels STEP which defines accidents as processes.
Problems Encountered in Development.
To develop realistic decision-
making training, the researchers selected STEP' s "mental movie" methodology as
the vehicle for helping aircraft pilots improve their understanding and use of
the decision process. Ideally, the mental movies used for decision making
training would replicate critical, real-world in-flight scenarios.
In the quest for scenarios, aviation accident reports were selected which had an apparently above average quantity and quality of concrete pilot decision making information. However, as attempts were made to analyze those reports with STEP's "mental movie' approach, the researchers quickly realized that the pilot decision steps required to produce the outcome were rarely reported in concrete, "pointed to" terms or words. The table below displays the presence (Y) or absence (N) of such data in a report where 10 potential pilot decision sets were identified.
These results were not exceptional. In five accident cases. a 51 potential pilot decision sets were identified. Yet analysis showed that only 3 of those 51 decision sets contained fully reported decision events. The data base of specific, concrete accident (or near miss) decision events needed to develop the kind of requested pilot decision making training was clearly not available.
Project Completion. The project was completed, assisted by informed
analysis and speculation, using the DECIDE model steps and the data available in the reports. The DECIDE step descriptions were refined and used in the training. The final model is shown below and is described more fully in the
project report to AOPA.
CONSEQUENCES OF THE LACK OF CONCRETE DATA
This research project highlights the lack of concrete accident event data. Even when accident reports are selected because of their apparent richness in concrete decision data, the reporting reflects an "attributes" view of decision making. Unfortunately, most accident data is reported in relatively abstract terminology, as attributes rather than as actions. Consider the two statements, "X was inattentive," and "X looked at wrong gauge." The first reflects a "state of being" or an attribute; the second, an observable, dynamic living action, an event. And it is in the real world of specific, concrete events that pilots make decisions and sometimes have accidents.
When the accident reports favoring attribute information are analyzed and plotted (using the STEP analytical procedures), the consequences of this attribute view become evident. One result is that the pilot's decision making process cannot be visualized or assessed in concrete terms. A second result is that ambiguities and gaps are found in accident event data. These two consequences combine to severely cripple attempts to provide concrete and specific pilot decision making training.
OVERVIEW OF STEP METHODOLOGY
The STEP methodology used to overcome the ambiguity of the abstract human factors data in this research seems generally applicable.
Documentation of Accident Events.
When STEP methodology in used in accident investigation, each specific "event," each pointed to occurrence, is documented in a disciplined Building Block format consisting, as minimum, of a uniquely named actor, a single observable action, and measurable. concrete descriptors.
Such documentation is replicable in that words on or near the bottom rungs of the ladder of abstraction are used, words whose specific meanings permit visualization of the actions. STEP' s disciplining methodology assures compatibility of event documentation, even when several investigators are attempting to describe the same accident process.
Displaying of Accident Events
After several Building Blocks have been documented, they are displayed on a worksheet, linked by arrows to indicate the precede follow relationships among them. Before a new Building Block is placed in the individual actor's row on the worksheet, it is tested against previously displayed Building
Blocks and placed in the appropriate time column indicating whether it occurred before, at the same time as, or after each previously displayed Building Block. Thus event sets are developed and documented. Each event set consists of at least two Building Blocks. In some cases, the event set may involve several Building Blocks, depending on whether the preceding event triggered several following events or whether more than one preceding event was necessary to trigger the following event. These are indicated by arrow links.
Testing Accident Documentation.
As a way of testing the logical consistency of events, linking arrows connections are examined to determine if (1) all the preceding events which were necessary to trigger the following event have been displayed and (2) if the preceding events were sufficient, by themselves, to trigger the following event. Finally, the event set is reviewed to determine whether too many events have been displayed. i. e. whether one or more of the following events could have been triggered by one fewer preceding event.
Identifying Gaps in Understanding of the Accident Process.
Gaps in the flow of events provide the investigators with hypotheses to aid the search for data. STEP' s BackSTEP technique is used to postulate events. In essence, backSTEP is a sequential logic tree diagram, placed horizontally to call attention to the importance of the relative timing of the displayed events. Using backSTEP, the investigator continually asks, "What could have triggered the event on the right?", until one of the events postulated as preceding the event on the right resembles one of the previously documented preceding events. The evidence of the specific intervening postulated events is then sought.
APPLICATION OF STEP TO PILOT DECISION MAKING RESEARCH
STEP Enables Visualization of the Decision Making Process
The objective, of the STEP methodology is to enable the researcher and anyone else to produce a mental movie of the scenario of interest. That mental movie should be complete and satisfying as well as capable of communicating the scenario being analyzed. Thus a STEP description of a pilot's actions during a critical time period should enable visualization of each decision set in terms of its constituent event(s). This, in turn, makes it possible to replicate and control the process outcomes in the future.
STEP Can Define Hypotheses for Investigation and Research
STEP' s requirements for concrete terminology and disciplined documentation of reported accident data enable development of a data base of decision sets. Direct comparison of decision sets would be possible because of the use of concrete words in the documentation of events, eliminating the need for interpretation of data.
Hypotheses in Investigations. Investigators develop hypotheses during investigations to identify areas where "research" is needed to produce accident data. STEP assists in the development of such hypotheses, first, by disciplining the display of decision events on STEP worksheets and, second, through the use of backSTEP.
Hypotheses in Research. As accident investigators report specific data in STEP format, the grist for the generation of more encompassing hypotheses by researchers becomes available, Individual displays of decision sets in the STEP format could he overlaid. Such overlaying could be expected to produce hypotheses in the form of "When event 1 occurs, what event(s) follow?" As a data base of decision sets is built, the ability to compare and analyze decision sets becomes increasingly reliable. Graphically, this analysis can be illustrated with this three dimensional figure which shows the way in which decision sets can be "stacked" to facilitate comparison of the processes.
STEP Can Be Used With Decision Making Models
Testing, the Model's Validity. As STEP worksheets of individual decision sets can be overlaid as a way of testing hypotheses, so can STEP analyses of decision making models be "stacked" with scenarios based on real world decision making experience. Comparison of the model to the documented experience provides a clear method of testing the validity of a decision making model.
Using the Model for Data Acquisition, Organization and Testing. 0nce a
decision making model has been validated by comparison to real world decision
making experiences, that model can then be used as a hypothesis to be
tested during the data acquisition phase of accident investigation. Having a
model would assist the investigator identify the data to be sought. Missing
information would be clearly identified. The thoroughness of the investigation could be assessed by comparison of a STEP worksheet of the accident
process to the validated model.
The figure below illustrates the use of STEP with the DECIDE model described earlier. Note how absent decision set events are easily identified.
Use of STEP based decision sets has two values. First, it can document the recurrence of specific decision making events. Second, it can assist the development, from observation and documentation of events, of hypotheses to be tested, both during the investigation of individual accidents and in research as comparabilities among decision sets become evident. Historically, human factor hypotheses have been developed from attribute rather than concrete data. As a result, these hypotheses tend to be high on the ladder of abstraction. This placement has clear implications for the response time required to get answers to identified problems. When the problem is clearly and concretely identified, as it is in STEP, answers are facilitated. On the other hand, when the problem is not clearly defined, several solutions are possible, depending on the interpretation each involved party gives to the abstract, ambiguous information.
With the STEP decision event set approach, each hypothesis, whether in an accident investigation or in the research environment, is positioned at a very low rung on that ladder, a rung at which the events involved are easily observed and quickly understood by all investigating strangers, researchers and decision makers (including those funding hardware changes and pilots walking to the flight line).
Clearly, use of the STEP decision event set methodology will not provide any immediate panaceas to resolve problems associated with pilot decision making. On the other hand, the STEP approach to learning more about decision making identifies problems that can be solved, just as the STEP methodology, when applied to the investigation of a single accident, identifies safety problems in need of high priority action to ameliorate risks.
At this point in our understanding of human behavior in risk situations, it may be that the best we can do is to direct attention to specific decision sets which, for at least one individual, can be documented as leading to an undesired following event. If a pilot can be told, with certainly, that, if event 1 occurs, unless events 2 through 9 are accomplished, a accident is certain, then a major contribution to the pleas of pilots for concrete decision making data will have been made.
The implications of the use of the STEP approach to the discovery, documentation and display of events constituting the decision process are several. First, and probably most significantly, it suggests a way by which human factor research can contribute to a better understanding, prediction and control of complex human responses to specific and definable scenarios. The STEP methodology proposed in this paper provides the ability to consider a vast array of precede follow event pairs. The requirement to aggregate bits rather than chunks of data results in the research being defined, manageable and explainable to investigating strangers because of its focus on lower, real world rungs of the ladder of abstraction, nearer the ability to point to what is being investigated.
Is this a "reductionist' approach to the investigation of the complexity of human performance? We think not. The same scientific principles as are at work in other scientific fields are at work with this approach. The difference lies in the techniques of hypothesis generation and testing and in the inclusiveness of the hypotheses themselves. This approach demands that hypotheses be generated that call for careful examination of concrete "observations" of events rather than abstract conclusions about phenomena. Parallels can be found in the chemist working with elements rather than with compounds, with the life scientist working with DNA molecules rather than with organisms.
As the life scientist focuses on a single phenomenon, so does human factors research need the discipline that STEP and the overlays described above provide. These focus the researchers' attention on the comparison of highly correlated decision sets so that all the implications can be examined. The necessity of verifying the congruence of decision sets from two separate situations of interest highlights the necessity of carefully gathering and documenting all data, especially event descriptors, carefully, concretely, and indicating data sources for verification of congruence. This also highlights the need for consciousness of the Ladder of Abstraction in preparing Building Blocks so that words are easily comprehended and visualized.
 Lawton. Russell. "The Decide Approach to Training Pilots in Pilot Decision Making." Presented to Third International Pilot Decision Making Conference, Ottawa. Canada. December 2 3. 1986.
. Rockwell. T., and Griffin, W. "An Investigation into Pilot and System Response to Critical In flight Events," National Air and Space Administration. Ames Research Center. Moffett Field, CA, 1981
. "Interim Phase I Report. Proposed Generalized Teaching Model of General Aviation Mishap Process and General Aviation Pilot Decision making Process." Events Analysis. Inc., Oakton. VA 1985.
. Hendrick. K., and Benner, L. Jr. Investigating Accidents Kith ST. Marcel Dekker. Inc., New York, NY. 1987.
. Hendrick and Benner, Investigating Accidents with STEP, Appendix H.