SAFETY & UNDERSTANDING:
DESIGN AND DEVELOPMENT OF INTELLIGENT TRAINING
Walter Lee and Paul Pangaro
PANGARO Incorporated
This paper was invited by the American Institute of Chemical Engineers and was presented at their annual meeting in 1994. It reports the work performed for a nuclear power client which was also published in "Including the Whys and Wherefores in Procedural Training: Intelligent Training for Emergencies in Nuclear Power Plant by Lee, Gander, Pangaro & Wilkinson, IEEE International Conference on Systems, Man and Cybernetics, Charlottesville, VA October '91. Although this paper was composed by me, it is based on the IEEE report of the project, composed by Lee. Lee, Gander, Wilkinson and myself were close collaborators throughout the project.
ABSTRACT
The authors describe the large-scale software system they
developed for the training of emergency procedures and they generalize
requirements for similar situations. Development of the Emergency
Operating Procedures Support System (EOP-SS), under contract
to the nuclear utility Niagara Mohawk Power Corporation, has
required innovations in the architecture and components of intelligent
training systems. The results are applicable to other situations
where safety is critical, system components and instrumentation
are vast, and emergency conditions are impossible to predict
in advance.
The cybernetic approach defines the interactions among practice
of procedures (including at extremes, far beyond safe conditions);
access to system descriptions (including design bases and theory
of operations); and feedback from experts to operator response
in simulated drills. The seamless linking of these activities
has required design of a sophisticated software archi-tecture
as well as innovations in components of the EOP-SS, specifically:
adaptive hypertext; the graphical definition of expert knowledge
that links purpose and action; and the capability for modification
of drills, system descriptions, and "correct re-sponse"
without programming changes (that is, the system is "data
driven").
BACKGROUND
PANGARO Incorporated, under contract to Niagara Mohawk Power Corporation (NMPC), has completed development of a large-scale software system for the training of nuclear power plant operators in emergency procedures.
Figure 1: Typical paper-based emergency
procedure 
Emergency operating procedures (EOPs) specify those conditions
outside of "normal" operations under which specific
responses are to be followed in order to minimize risks to safety.
EOPs are complex to learn and interpret, because of their range
and large number of details. In the normal course of training
and operations, operators get comparatively little opportunity
to practice running the plant or its simulator at the extremes
of its safe operating conditions. Although there are some special
considerations in regard to nuclear power regulation and federal
law regarding proper execution of EOPs, the problems and solutions
involved are precisely those of complex industrial plants where
safety is critical and operator understanding of plant behavior
are key.
SAFETY TRAINING
Two modes of training are already available to the training departments of nuclear utilities, but on review during early phases of our relationship with NMPC we learned that there is too little opportunity for operators to:
- explore scenarios to the limits of their curiosity, responsibilities and knowledge
- experiment freely as individuals, along paths that might be construed as unsafe or inappropriate, to understand the implications
- take the plant, at least in simulation, far into off-normal operations to enhance their under-standing and experience of the plant in unusual circumstances.
The desire to provide a means for operators of complex plants
to explore in these modes may seem an unnecessary luxury or of
secondary interest to safe operations, but we simply point out
that the purposes listed above are precisely those for flight
simulators in the aircraft industry. No one questions the application
of that training technology to significantly (and cost-effectively)
improve the safety of aircraft operations. We proposed the enhancement
of pre-existing training capabilities of NMPC, already consisting
only of the modes of classroom and simulator, as characterized
by the "training triangle" in Figure
2.
Figure 2:
The pre-existing training modes, Classroom and Simulation,
and their complement, the EOP-SS.
This was funded by the R&D department in a 3rd phase of relationship
with NMPC, resulting in full prototype development of the Emergency
Operating Procedures Support System, the EOP-SS.
APPROACH
We set about to create the complementary capability represented
in the training triangle diagram by considering design requirements
for the EOP-SS. A number of large and individually-complex subsystems
were required to provide a useful, efficient and powerful capability
in a stand-alone software/hardware environment. An important
requirement was that the system would be maintainable by site
personnel, without requiring a return to programming in order
to maintain a completely up-to-date, as well as integrated and
accepted, part of daily activities of the plant. Thus some pieces
of the EOP-SS would be used by operators to interact with and
learn from, while others would be used by NMPC's EOP experts
to depict EOPs and to embed their knowledge of correct EOP actions
and their technical bases.
We began our design process recognizing the pre-existing world-view
of plant operators, and the world of emergency operations generally.
An immediate goal was the ability to view and manipulate the
pre-existing representation of the EOPs on-line in the training
system, as a common focal point of both the operators and the
EOP-SS.
The image of Figure 3 is directly from
the software interface of the EOP-SS, showing a portion of EOP-3
(albeit far smaller than the actual 19" monitor shows).
The operator is at liberty to pan around and zoom into each EOP,
which on paper require at least 21" by 32" sheets to
contain, and even then at less-than-perfectly-clear quality.
The image quality on-screen is exceptionally fine at 1000 lines,
and the visual match to the paper EOPs is precise.
Figure 3: An EOP as displayed
in the EOP-SS.
The existence of the on-line EOPs provides the means for an operator
to point at a particular EOP step and to have the EOP-SS know
the operator's focus of attention exactly. This is critical for
allowing the operator to direct the attention of the system to
a particular place in the EOPs and to ask questions; for example,
to obtain the technical bases for a step, or ask about specific
actions to be taken, or review particular plant status that relates
to the step. Conversely, by blinking part of the EOP flowchart,
the EOP-SS can draw the operator's focus of attention to a particular
step, and thereby indicate, given current plant conditions, what
area(s) is(are) currently active, or what step(s) should be generating
specific action(s) in response to plant conditions.
PLANT CONTROL
Given the EOPs as the primary constraint of the training domain, a study was performed by the originators of the EOPs for NMPC to specify all plant controls (inputs) and instrumentation (outputs) that would be required to exercise the EOPs. That taxonomy was the basis for the design of control screens that an operator would manipulate at a graphical user interface (GUI) to control the plant. An accurate, thermal-hydraulic model of a nuclear plant with real-time control of all parameters relevant to the exercise of EOPs had to be made available to operators for simulated emergency scenarios. The basis of such a software simulation was also purchased from consultant experts. This software required some modifications to provide all the necessary EOP controls, and to tailor specific parameters to mimic the operation of the specific plant, Nine Mile One. This was made available in FORTRAN and the state-of-the-art PC of the day was purchased (the first-available 25 Mhz -486) to execute the simulation. Since this required a dedicated processor, and because other, more powerful software development environments were available and more suitable to the construction of the entire training environment, all other programming was done in the most powerful software development engine available, that of the AI workstations designed and constructed by Symbolics, Inc. With two large, 1000-line 19" monitors, one in color and one in black and white, the system is rich in interaction and response.
Figure
4: One of six screens for operator control of plant
and instrumentation read-outs for emergency conditions.
Six separate software screens, of which Figure
4 is one example, provided access to all controls for input,
and instrument readings for output, of plant conditions required
to execute EOPs.
SCENARIO MODE
The primary mode of use of the EOP-SS is, naturally enough, to allow the operator to run through simulated emergency scenarios and practice the EOPs, just like in the full-scope, high-fidelity simulator. But simple practice, especially in an un-supervised mode (which, as per the training triangle, was an important complement to pre-existing training modes) is far less valuable without some means to track operator performance, and to provide detailed and accurate feedback about it. Taking advantage of the popular acceptance of expert systems within the training context ("intelligent training systems", that is, the marriage of training and expert systems) we introduced the notion of an "EOP Coach." During a drill, the actions of the operator are recorded, along with the "correct" actions of the software Coach, as per the schematic structure of Figure 5.
Figure 5:
Structural schematic of EOP-SS during operator drill.
While the plant is taken through emergency scenarios, both operator
actions and expert "EOP Coach" proposed actions are
stored. Operator actions also control plant conditions and modify
the outcome of the scenario.
EOP COACH
The value of the EOP Coach is directly related to its accuracy
and appropriateness. Accuracy, in that it must detect specific
plant conditions and generate specific actions that are required
by the written, flowchart-based EOPs, just as they would be interpreted
by an EOP expert under the same conditions. Appropriateness,
in that although a totally strict interpretation of a computer-defined
procedure might produce a particular action, an intelligent human
operator might never make such an action, given a typical scope
of awareness and experience.
In practice such demands on expert systems are very high. Because
of those demands a decision was made early on to provide directly
to the plant experts the capability for definition, modification
and maintenance of the Coach's expertise. This avoids the traditional
problems of inserting the "knowledge engineer" into
the loop of programming the expert's knowledge of EOPs into the
Coach. This was particularly important because the EOPs themselves
were subject to occasional but persistent revisions, making return
to software programming to update the EOP-SS clumsy and subject
to greater problems in verification and validation. This was
achieved by crucial features of the software:
- All "rules" are defined through a graphical user interface that could be operated by plant experts (Figure 6). Every instrument and control required for EOP response was made available on menus that were displayed only when the particular syntax of the rule being defined allowed it.
Figure 6: Graphical user interface for defining and maintaining Coach knowledge. These structures are directly manipulated via menus to create the "rules" of the expert system that make up the Coach. No conventional programming is required to specify all EOP responses.
- The appearance of the EOPs on screen, tied as they must be to the appearance of the EOPs on paper originals, must also be subject to easy update. Again a GUI was developed that would provide for the swift and easy editing of the on-line images of the EOPs (Figure 7).
Figure 7: Preparation
of the on-line EOPs is done entirely through another graphical
user interface. Updates to the plant EOPs can easily be propagated
to the EOP-SS by plant personnel.
OPERATOR COACHING
Up to this point we have described the means for an operator
to interact with an accurate simulation of plant activity in
emergency scenarios, through a software interface providing plant
controls for input and plant instruments for reading plant state.
The operator's actions during such a drill are recorded, while
at the same time an expert EOP Coach is observing the same plant
states and its actions are also being recorded (Figure
5). As shown in Figure 8, the complementary
mode of the EOP-SS is playback of the drill, just like a human
expert's critique after a simulator drill.
Figure 8:
Schematic structure of EOP-SS during drill playback and coaching.
The results of both operator actions and proposed EOP Coach actions
are compared and the results tabulated on a support screen for
review by the operator.
The same scenario to which the operator was subjected again controls
the plant state. The previously-recorded drill results from both
the EOP Coach and the operator are compared and the results displayed
to the operator.
The operator must be placed in a context for receiving feedback
from the Coach that is both clear and non-confrontational. The
latter is achieved in part by careful wording. For example, the
notion of "error" was considered an inappro-priate
way of referring to any operator action, especially when being
compared to a mechanical, computer-generated Coach. Instead the
term "discrepancy" was used. Similar care was taken
by listening to operator input during the entire design process,
input considered crucial to the acceptance of any such system
to its end-users.
Clarity in the Coach is achieved by a series of carefully designed
screens that both display the results of comparing operator's
actions to the Coach's, as well as provide detailed access to
the reasoning behind the Coach's decisions, all completely under
operator control. Figure 9 is the first
of such screens, showing a synopsis of the results of the drill.
Figure 10 provides more detailed reasoning
"behind" the action. Finally Figure
11 shows how the operator can move directly from the Coach's
reasoning to the EOP (procedure) itself. [Click on each Figure
for additional explanations.]
Figure 9:
First EOP Coach screen, showing initial comparison of operator's
and Coach's actions given the same plant conditions.
Figure
10: Detailed Coach reasoning behind a specific action,
displayed in response to operator's specific request about that
action.
Figure
11: Display resulting from operator clicking on EOP
step name in explanation of EOP Coach's chosen action (Figure
10).
Because of the dynamic nature of the EOP-SS and its many modes of interaction with the operator, it is difficult to provide a representative description of the richness and effectiveness of its use.
TECHNICAL BASES
It is our belief and experience, supported by the literature
of learning theory, that proper and comprehensive understanding
of systems of the complexity of industrial plants requires more
than exercise of procedures, no matter how extensive. It is critical
that operators possess deep understanding of the reasons why
plant conditions occur and how mitigating actions are effective.
Otherwise unforeseen situations, which invariably occur, cannot
be handled intelligently (a look at the history of industrial
accidents reaffirms this).
Exercising procedures can and must be part of training, but more
is needed. Explanations behind, and purposes for, those procedures
must support the drills in which they are applied.
Our response to this need was to integrate a sophisticated, large-scale
training environment designed and developed for other clients
of ours who are also concerned with complex decision-making situations.
Called THOUGHTSTICKER, this system provides a complete hypertext
authoring and delivery environment for capturing and displaying
all of the Technical Bases on which the design and construction
of the plant is founded. Far more can be said about the innovations
of THOUGHTSTICKER, and these descriptions are available in a
separate document.
Figure 12:
Display of Technical Bases in an operator-driven interface, incorporating
previously-developed features from an adap-tive hypermedia system
called THOUGHTSTICKER. The EOP-SS remembers what explanations
have been seen before by each individual operator and picks subsequent
explanations appropriately.
DEVELOPING EMERGENCY PROCEDURES
The EOP-SS has all the required elements for the development of emergency procedures. In practice the EOP-SS could provide considerable advantages over the usual means wherein the design bases of the plant are used in serial simulations. This is easily accomplished by using the EOP Coach not simply to generate actions for recording, but to control the plant simulation.
Figure
13: The configuration of the EOP-SS as a tool for
developing emergency procedures. Experi-ments can be performed
and statistics recorded of the results of differing emergency
mitigation strategies.
As per the schematic configuration of the EOP-SS in Figure 13, multiple instances of the plant model can be executed under the control of alternative EOP models. Comparisons can be made of the consequences of their alternative responses to emergencies, and their relative merits can easily be measured. Metrics of length of time of excursion into off-normal conditions, and depth of excursion away from safe operations (for example, degree of coolant loss), can easily be tracked for many variations of emergency scenarios and EOP response.
SUMMARY
The EOP-SS constitutes the first of its kind: a stand-alone,
one-on-one training system for complex decision making designed
for training for plant safety. The system is data-driven, making
all maintenance and modifications available to plant personnel
without programmer intervention. The individual subsystems (simulation,
plant controller, EOP Coach, adaptive hypertext, EOP rule-definition
interface, EOP display-constructor interface) make for a seamless
entirety that is easy for plant personnel to learn.
Most importantly for the operator/trainee, this seamless architecture
allows for complete flexibility in examining any aspect of emergency
mitigation strategies. Starting from any representation (EOP
step, Technical Bases, Coach or operator action), the operator
can move to any other representation by a single mouse gesture.
This unique capability provides for a training experience in
the cognitive domain of the "hows and whys" of plant
operations and safety procedures. The EOP-SS makes available
to an individual operator in his/her specific context of an emergency
scenario not just what action should be taken, but also how the
action implements the EOP to mitigate the emergency, and why
the system responds as it does. This is analogous to, but a powerful
extension of, the simple "whats" of a simulator, whether
of an aircraft or industrial plant. By providing the comparison
of experts' behaviors for compari-son to one's own, as well as
reasons for such behaviors delivered in a manner that is adaptive
to an individual's experiences with the training, the operator's
understanding of emergency mitigation strategies is as great
as possible, short of direct experiences in real emergencies,
experiences we would all prefer to avoid. Application of the
architecture and concepts behind the EOP-SS can help avoid them.
-end-
For further details of the background on EOPs,
and the design bases of the EOP-SS, see "Including the Whys
and Where-fores in Procedural Training: Intelligent Training
for Emergencies in Nuclear Power Plants", by Lee, Gander,
Pangaro and Wilkinson, IEEE International Conference on Systems,
Man and Cybernetics, Charlottesville, VA ,October 1991.
© Copyright Paul Pangaro 1994 - 2000. All Rights Reserved.