• Pangaro Incorporated




    Walter Lee and Paul Pangaro
    PANGARO Incorporated

    This paper was invited by the American Institute of Chemical Engineers and was presented at their annual meeting in 1994. It reports the work performed for a nuclear power client which was also published in "Including the Whys and Wherefores in Procedural Training: Intelligent Training for Emergencies in Nuclear Power Plant by Lee, Gander, Pangaro & Wilkinson, IEEE International Conference on Systems, Man and Cybernetics, Charlottesville, VA October '91. Although this paper was composed by me, it is based on the IEEE report of the project, composed by Lee. Lee, Gander, Wilkinson and myself were close collaborators throughout the project.


    The authors describe the large-scale software system they developed for the training of emergency procedures and they generalize requirements for similar situations. Development of the Emergency Operating Procedures Support System (EOP-SS), under contract to the nuclear utility Niagara Mohawk Power Corporation, has required innovations in the architecture and components of intelligent training systems. The results are applicable to other situations where safety is critical, system components and instrumentation are vast, and emergency conditions are impossible to predict in advance.

    The cybernetic approach defines the interactions among practice of procedures (including at extremes, far beyond safe conditions); access to system descriptions (including design bases and theory of operations); and feedback from experts to operator response in simulated drills. The seamless linking of these activities has required design of a sophisticated software archi-tecture as well as innovations in components of the EOP-SS, specifically: adaptive hypertext; the graphical definition of expert knowledge that links purpose and action; and the capability for modification of drills, system descriptions, and "correct re-sponse" without programming changes (that is, the system is "data driven").


    PANGARO Incorporated, under contract to Niagara Mohawk Power Corporation (NMPC), has completed development of a large-scale software system for the training of nuclear power plant operators in emergency procedures.

    Figure 1: Typical paper-based emergency procedure

    Emergency operating procedures (EOPs) specify those conditions outside of "normal" operations under which specific responses are to be followed in order to minimize risks to safety. EOPs are complex to learn and interpret, because of their range and large number of details. In the normal course of training and operations, operators get comparatively little opportunity to practice running the plant or its simulator at the extremes of its safe operating conditions. Although there are some special considerations in regard to nuclear power regulation and federal law regarding proper execution of EOPs, the problems and solutions involved are precisely those of complex industrial plants where safety is critical and operator understanding of plant behavior are key.


    Two modes of training are already available to the training departments of nuclear utilities, but on review during early phases of our relationship with NMPC we learned that there is too little opportunity for operators to:

    • explore scenarios to the limits of their curiosity, responsibilities and knowledge
    • experiment freely as individuals, along paths that might be construed as unsafe or inappropriate, to understand the implications
    • take the plant, at least in simulation, far into off-normal operations to enhance their under-standing and experience of the plant in unusual circumstances.

    The desire to provide a means for operators of complex plants to explore in these modes may seem an unnecessary luxury or of secondary interest to safe operations, but we simply point out that the purposes listed above are precisely those for flight simulators in the aircraft industry. No one questions the application of that training technology to significantly (and cost-effectively) improve the safety of aircraft operations. We proposed the enhancement of pre-existing training capabilities of NMPC, already consisting only of the modes of classroom and simulator, as characterized by the "training triangle" in Figure 2.

    Figure 2: The pre-existing training modes, Classroom and Simulation, and their complement, the EOP-SS.

    This was funded by the R&D department in a 3rd phase of relationship with NMPC, resulting in full prototype development of the Emergency Operating Procedures Support System, the EOP-SS.


    We set about to create the complementary capability represented in the training triangle diagram by considering design requirements for the EOP-SS. A number of large and individually-complex subsystems were required to provide a useful, efficient and powerful capability in a stand-alone software/hardware environment. An important requirement was that the system would be maintainable by site personnel, without requiring a return to programming in order to maintain a completely up-to-date, as well as integrated and accepted, part of daily activities of the plant. Thus some pieces of the EOP-SS would be used by operators to interact with and learn from, while others would be used by NMPC's EOP experts to depict EOPs and to embed their knowledge of correct EOP actions and their technical bases.

    We began our design process recognizing the pre-existing world-view of plant operators, and the world of emergency operations generally. An immediate goal was the ability to view and manipulate the pre-existing representation of the EOPs on-line in the training system, as a common focal point of both the operators and the EOP-SS.

    The image of Figure 3 is directly from the software interface of the EOP-SS, showing a portion of EOP-3 (albeit far smaller than the actual 19" monitor shows). The operator is at liberty to pan around and zoom into each EOP, which on paper require at least 21" by 32" sheets to contain, and even then at less-than-perfectly-clear quality. The image quality on-screen is exceptionally fine at 1000 lines, and the visual match to the paper EOPs is precise.

    Figure 3: An EOP as displayed in the EOP-SS.

    The existence of the on-line EOPs provides the means for an operator to point at a particular EOP step and to have the EOP-SS know the operator's focus of attention exactly. This is critical for allowing the operator to direct the attention of the system to a particular place in the EOPs and to ask questions; for example, to obtain the technical bases for a step, or ask about specific actions to be taken, or review particular plant status that relates to the step. Conversely, by blinking part of the EOP flowchart, the EOP-SS can draw the operator's focus of attention to a particular step, and thereby indicate, given current plant conditions, what area(s) is(are) currently active, or what step(s) should be generating specific action(s) in response to plant conditions.


    Given the EOPs as the primary constraint of the training domain, a study was performed by the originators of the EOPs for NMPC to specify all plant controls (inputs) and instrumentation (outputs) that would be required to exercise the EOPs. That taxonomy was the basis for the design of control screens that an operator would manipulate at a graphical user interface (GUI) to control the plant. An accurate, thermal-hydraulic model of a nuclear plant with real-time control of all parameters relevant to the exercise of EOPs had to be made available to operators for simulated emergency scenarios. The basis of such a software simulation was also purchased from consultant experts. This software required some modifications to provide all the necessary EOP controls, and to tailor specific parameters to mimic the operation of the specific plant, Nine Mile One. This was made available in FORTRAN and the state-of-the-art PC of the day was purchased (the first-available 25 Mhz -486) to execute the simulation. Since this required a dedicated processor, and because other, more powerful software development environments were available and more suitable to the construction of the entire training environment, all other programming was done in the most powerful software development engine available, that of the AI workstations designed and constructed by Symbolics, Inc. With two large, 1000-line 19" monitors, one in color and one in black and white, the system is rich in interaction and response.

    Figure 4: One of six screens for operator control of plant and instrumentation read-outs for emergency conditions.

    Six separate software screens, of which Figure 4 is one example, provided access to all controls for input, and instrument readings for output, of plant conditions required to execute EOPs.


    The primary mode of use of the EOP-SS is, naturally enough, to allow the operator to run through simulated emergency scenarios and practice the EOPs, just like in the full-scope, high-fidelity simulator. But simple practice, especially in an un-supervised mode (which, as per the training triangle, was an important complement to pre-existing training modes) is far less valuable without some means to track operator performance, and to provide detailed and accurate feedback about it. Taking advantage of the popular acceptance of expert systems within the training context ("intelligent training systems", that is, the marriage of training and expert systems) we introduced the notion of an "EOP Coach." During a drill, the actions of the operator are recorded, along with the "correct" actions of the software Coach, as per the schematic structure of Figure 5.

    Figure 5: Structural schematic of EOP-SS during operator drill. While the plant is taken through emergency scenarios, both operator actions and expert "EOP Coach" proposed actions are stored. Operator actions also control plant conditions and modify the outcome of the scenario.


    The value of the EOP Coach is directly related to its accuracy and appropriateness. Accuracy, in that it must detect specific plant conditions and generate specific actions that are required by the written, flowchart-based EOPs, just as they would be interpreted by an EOP expert under the same conditions. Appropriateness, in that although a totally strict interpretation of a computer-defined procedure might produce a particular action, an intelligent human operator might never make such an action, given a typical scope of awareness and experience.

    In practice such demands on expert systems are very high. Because of those demands a decision was made early on to provide directly to the plant experts the capability for definition, modification and maintenance of the Coach's expertise. This avoids the traditional problems of inserting the "knowledge engineer" into the loop of programming the expert's knowledge of EOPs into the Coach. This was particularly important because the EOPs themselves were subject to occasional but persistent revisions, making return to software programming to update the EOP-SS clumsy and subject to greater problems in verification and validation. This was achieved by crucial features of the software:

    • All "rules" are defined through a graphical user interface that could be operated by plant experts (Figure 6). Every instrument and control required for EOP response was made available on menus that were displayed only when the particular syntax of the rule being defined allowed it.

    Figure 6: Graphical user interface for defining and maintaining Coach knowledge. These structures are directly manipulated via menus to create the "rules" of the expert system that make up the Coach. No conventional programming is required to specify all EOP responses.

    • The appearance of the EOPs on screen, tied as they must be to the appearance of the EOPs on paper originals, must also be subject to easy update. Again a GUI was developed that would provide for the swift and easy editing of the on-line images of the EOPs (Figure 7).

    Figure 7: Preparation of the on-line EOPs is done entirely through another graphical user interface. Updates to the plant EOPs can easily be propagated to the EOP-SS by plant personnel.


    Up to this point we have described the means for an operator to interact with an accurate simulation of plant activity in emergency scenarios, through a software interface providing plant controls for input and plant instruments for reading plant state. The operator's actions during such a drill are recorded, while at the same time an expert EOP Coach is observing the same plant states and its actions are also being recorded (Figure 5). As shown in Figure 8, the complementary mode of the EOP-SS is playback of the drill, just like a human expert's critique after a simulator drill.

    Figure 8: Schematic structure of EOP-SS during drill playback and coaching. The results of both operator actions and proposed EOP Coach actions are compared and the results tabulated on a support screen for review by the operator.

    The same scenario to which the operator was subjected again controls the plant state. The previously-recorded drill results from both the EOP Coach and the operator are compared and the results displayed to the operator.

    The operator must be placed in a context for receiving feedback from the Coach that is both clear and non-confrontational. The latter is achieved in part by careful wording. For example, the notion of "error" was considered an inappro-priate way of referring to any operator action, especially when being compared to a mechanical, computer-generated Coach. Instead the term "discrepancy" was used. Similar care was taken by listening to operator input during the entire design process, input considered crucial to the acceptance of any such system to its end-users.

    Clarity in the Coach is achieved by a series of carefully designed screens that both display the results of comparing operator's actions to the Coach's, as well as provide detailed access to the reasoning behind the Coach's decisions, all completely under operator control. Figure 9 is the first of such screens, showing a synopsis of the results of the drill. Figure 10 provides more detailed reasoning "behind" the action. Finally Figure 11 shows how the operator can move directly from the Coach's reasoning to the EOP (procedure) itself. [Click on each Figure for additional explanations.]

    Figure 9: First EOP Coach screen, showing initial comparison of operator's and Coach's actions given the same plant conditions.

    Figure 10: Detailed Coach reasoning behind a specific action, displayed in response to operator's specific request about that action.

    Figure 11: Display resulting from operator clicking on EOP step name in explanation of EOP Coach's chosen action (Figure 10).

    Because of the dynamic nature of the EOP-SS and its many modes of interaction with the operator, it is difficult to provide a representative description of the richness and effectiveness of its use.


    It is our belief and experience, supported by the literature of learning theory, that proper and comprehensive understanding of systems of the complexity of industrial plants requires more than exercise of procedures, no matter how extensive. It is critical that operators possess deep understanding of the reasons why plant conditions occur and how mitigating actions are effective. Otherwise unforeseen situations, which invariably occur, cannot be handled intelligently (a look at the history of industrial accidents reaffirms this).

    Exercising procedures can and must be part of training, but more is needed. Explanations behind, and purposes for, those procedures must support the drills in which they are applied.
    Our response to this need was to integrate a sophisticated, large-scale training environment designed and developed for other clients of ours who are also concerned with complex decision-making situations. Called THOUGHTSTICKER, this system provides a complete hypertext authoring and delivery environment for capturing and displaying all of the Technical Bases on which the design and construction of the plant is founded. Far more can be said about the innovations of THOUGHTSTICKER, and these descriptions are available in a separate document.

    Figure 12: Display of Technical Bases in an operator-driven interface, incorporating previously-developed features from an adap-tive hypermedia system called THOUGHTSTICKER. The EOP-SS remembers what explanations have been seen before by each individual operator and picks subsequent explanations appropriately.

    The EOP-SS has all the required elements for the development of emergency procedures. In practice the EOP-SS could provide considerable advantages over the usual means wherein the design bases of the plant are used in serial simulations. This is easily accomplished by using the EOP Coach not simply to generate actions for recording, but to control the plant simulation.

    Figure 13: The configuration of the EOP-SS as a tool for developing emergency procedures. Experi-ments can be performed and statistics recorded of the results of differing emergency mitigation strategies.

    As per the schematic configuration of the EOP-SS in Figure 13, multiple instances of the plant model can be executed under the control of alternative EOP models. Comparisons can be made of the consequences of their alternative responses to emergencies, and their relative merits can easily be measured. Metrics of length of time of excursion into off-normal conditions, and depth of excursion away from safe operations (for example, degree of coolant loss), can easily be tracked for many variations of emergency scenarios and EOP response.


    The EOP-SS constitutes the first of its kind: a stand-alone, one-on-one training system for complex decision making designed for training for plant safety. The system is data-driven, making all maintenance and modifications available to plant personnel without programmer intervention. The individual subsystems (simulation, plant controller, EOP Coach, adaptive hypertext, EOP rule-definition interface, EOP display-constructor interface) make for a seamless entirety that is easy for plant personnel to learn.

    Most importantly for the operator/trainee, this seamless architecture allows for complete flexibility in examining any aspect of emergency mitigation strategies. Starting from any representation (EOP step, Technical Bases, Coach or operator action), the operator can move to any other representation by a single mouse gesture. This unique capability provides for a training experience in the cognitive domain of the "hows and whys" of plant operations and safety procedures. The EOP-SS makes available to an individual operator in his/her specific context of an emergency scenario not just what action should be taken, but also how the action implements the EOP to mitigate the emergency, and why the system responds as it does. This is analogous to, but a powerful extension of, the simple "whats" of a simulator, whether of an aircraft or industrial plant. By providing the comparison of experts' behaviors for compari-son to one's own, as well as reasons for such behaviors delivered in a manner that is adaptive to an individual's experiences with the training, the operator's understanding of emergency mitigation strategies is as great as possible, short of direct experiences in real emergencies, experiences we would all prefer to avoid. Application of the architecture and concepts behind the EOP-SS can help avoid them.


    For further details of the background on EOPs, and the design bases of the EOP-SS, see "Including the Whys and Where-fores in Procedural Training: Intelligent Training for Emergencies in Nuclear Power Plants", by Lee, Gander, Pangaro and Wilkinson, IEEE International Conference on Systems, Man and Cybernetics, Charlottesville, VA ,October 1991.

    © Copyright Paul Pangaro 1994 - 2000. All Rights Reserved.