ORR/RA Lessons Learned, Summarized from DOE/NNSA Documentation and References

Achieving Readiness

The successful completion of any ORR will be strongly influenced by the degree to which readiness to start program work has been achieved prior to the start of the ORR. While it is the responsibility of line management to achieve a condition of readiness, experience indicates that the ORR Team Leader can influence the success of the process to gain readiness during his/her preparations for the ORR. In many cases, the potential for the particular problem to occur will become evident during pre-visits or other ORR preparations. The following lessons learned reflect areas in which the ORR Team Leader and team members may improve the success of the ORR.

Experience shows that prerequisites should provide significant detail and be fully measurable in order to permit line management to track each prerequisite to completion. The ORR/RA standard stresses the fact that the prerequisites should be tied to the Core Requirements, which are what will be evaluated in the readiness determination. So, ensure that the ORR Prerequisites specified in the Plan of Action support achieving readiness.

A thorough Management Self-Assessment (although not required by DOE O 425.1C) to assist line management in verifying that readiness has been achieved is an important final step in preparing for the ORR. The Contractor ORR Implementation Plan should reflect an intention to review the results of the Management Self-Assessment. The DOE ORR will review the results of the Contractor ORR. Sufficient time between the contractor and DOE ORRs should be planned to allow for finalization of documentation related to the closure and verification of the contractor ORR pre-start findings.

Inadequate, incomplete, or undefined incorporation of the safety basis documentation into procedures and policies has frequently resulted in delays in starting an ORR or in significant findings during the ORR. This is frequently caused by late development and approval of the safety basis documentation. Line management may not fully grasp the time and effort necessary to develop and implement all of the "flow down" procedures and surveillances necessary to put the requirements of the approved safety documentation into practice. Ensure full implementation of the safety basis.

Inadequate or incomplete evidence files or other verifiable documentation that demonstrate the prerequisite conditions have been met may result in an inability to verify readiness has been achieved. The Implementation Plan should clearly indicate the intentions to review these evidence files.

A responsible representative of the line management must be prepared to demonstrate to the ORR team that the conditions required to be met as a prerequisite to resumption, have actually been met. In most cases, this individual will be the counterpart who is assigned to work with the individual ORR Team member. The counterparts should be identified and utilized during the pre-visit.

Assigned counterparts must understand their responsibility to support the ORR team member.

The counterpart must present the information and documentation requested. The counterparts must accept responsibility for gaining information and resolving questions. The counterparts should be fully dedicated to a single ORR team member for the duration of the ORR field work. This expectation should be clearly defined during the pre-visit.

Lack of experience on the part of line management as to the expectations of the ORR team in the areas of drills and evolutions expected to occur during the ORR will lead to difficulty in completing established objectives. The Team Leader and team member must devote considerable attention to communicating expectations to responsible line management at the facility level.

Inadequate validation and verification of operational or maintenance/surveillance procedures that are newly prepared or recently modified will result in readiness not being achieved when the ORR starts. An experienced ORR team leader or team member may identify concerns with the adequacy and maturity of these procedures during the pre-visit. If problems are apparent, they should be identified to senior management at the end of the pre-visit.

Lack of formal, structured preparation by DOE line management of the personnel, programs, and documentation to be evaluated during the ORR will delay completion of required activities. DOE line management should be encouraged to conduct a Management Self-Assessment to ensure that prerequisite conditions for which they are responsible have been met. Potential problems in this area may be apparent during the pre-visit and, if noted, pointed out to senior DOE management at that time.

When scheduling sequential ORRs at the same site, management should ensure that delays associated with the first ORR do not negatively impact resources and readiness with respect to conducting the second one. Note: It should be apparent to the observant Team Leader during the pre-visits and other ORR preparations if difficulties with achieving readiness will be problematic. Although the ORR Team Leader will not be able to deal with the problems directly, his identification of the problems to the appropriate DOE or contractor line manager should be considered.

Items to Note Before Proceeding with an ORR

Thoroughly review the ORR Plan of Action, which, among other things, defines the scope of the ORR and lists the prerequisites for the ORR. The success of the ORR will be enhanced if the POA provides specific detail as to the scope of each core requirement, and defines the prerequisites in sufficient detail. Prerequisites should be specific and define measurable conditions. If the ORR Team Leader is identified prior to final approval of the POA, he should be included in the POA review and approval process and should ensure that the POA contains adequate detail to support a successful ORR. The POA should be approved as early as possible in the startup/restart process.

Gain familiarity with the facility and the project programs prior to the start of the ORR. Site access training, facility walkthroughs, and document reviews are essential for team members to gain the necessary familiarity with the project prior to the kickoff of the ORR.

Unescorted and unrestricted access to the facility is essential to effectively complete the review. This will require some investment of time and money in the preparation process. Get agreement, during development of the ORR plans, between the facility contractor and the Department of Energy on the details of the operations that are available for demonstration. Cold runs, partial simulations, and full walkthroughs are options, but an effort must be made to achieve a demonstration as close as possible to actual operations as allowed by the current restrictions of the shutdown.

Obtain and review the documented results of the contractor ORR/RA, including corrective action plans and evidence files documenting corrective action taken. Copies of corrective action documentation need to be readily available to the DOE ORR/RA team. Corrective action closure packages should be prepared in accordance with DOE-STD-3006-2000, Section 5.7.3.

Give consideration to requiring the contractor to deliver a completed set of surveillance procedures and authorization basis documents to the team leader as a prerequisite to the ORR/RA.

Ensure that records, plans, and other documentation requested by the ORR/RA team are readily available, preferably in a central location. A review several days in advance of the ORR/RA should be used to verify that what was requested is really there. In addition, this will reinforce the importance of the information requests by the individual team members.

When compliance is measured against local/contractor instructions, it is important for team members to verify that these local instructions adequately capture the requirements of higher-level documents.

Administration of the Results

Conduct a training session prior to the ORR/RA or no later than the beginning of the review process to communicate the expectations and deliverables required of the team. Format, content, and style (how to use and fill out) of Forms 1 (Assessment Forms), of Forms 2 (Deficiency Forms), and functional area summaries should be discussed to eliminate a compounding of the administrative load as the review progresses. Acceptable samples of these deliverables should be provided to all the team members prior to the start of the review. This handbook provides explicit instructions on the use of the forms, and the team leader can use this to provide training to the team. The technical editor or the Review Coordinator should be equally aware of the requirements and could conduct the training.

Advise site personnel as soon as an issue starts to be identified. The use of a draft Form 2 may be a method to clearly state the issue. Be sure that it is understood that the forms are draft documents (and should be marked as such) to avoid confusion or drawing of premature conclusions. Providing these forms to the site provides them an opportunity to produce further information that may clarify the issue and creates an air of openness that will contribute favorably to the review.

Conduct of Drills and Other Operations

Specify requirements for Emergency Preparedness or facility drills prior to the start of the ORR/RA. Detailed guidance on drill expectations, e.g., type of drill, desired preconditions, scenario requirements, etc., should be provided to the facility prior to the ORR/RA.

Ensure that site personnel understand fully the conditions that needed to be simulated during each drill and performance demonstration. All documentation such as Radiological Work Permits, Work Procedures, etc., must reflect the simulated condition.

Define, during the preparation phase, the role, authority and responsibilities of the ORR team when it is monitoring the conduct of an emergency exercise or drill. The ORR team must not disrupt the flow of the exercise. The ORR team is evaluating both the facility and the emergency preparedness training organization and drill control. These roles, authority and responsibilities require definition and must be fully understood by the facility managers.

Evaluate whether the contractor ORR/RA ensures the integration of equipment operability, procedure viability, and training of the operators in a performance-based methodology. Additionally, a detailed plan for the progression from completion of the ORR/RA to startup should be verified if a significant difference exists between the operations that can be demonstrated and normal operations.

Operations with multiple systems or which cover separate physical areas are better assessed when evaluated or divided into separate systems or areas and assigned to multiple team members.

ORR team members need to be briefed, as a group, on any Radiation Work Permits (RWPs) that they will be working to while conducting the ORR. Pre-briefing the RWPs avoids potential violations of RWP requirements by team members and saves valuable time when entering Material Access Areas for observation of work evolutions.

Logistics and Support

Communicate logistical needs and requirements to the site well in advance of the review. For example:

Specify what documents need to be available to the team and that these documents need to be in a central location for easy access by the team during the review.

Specify that computers need to be up-to-date models. Printers and copying machines, with adequate supplies of paper, toner, etc., need to be dedicated to the team for the duration of the review.

Specify the word processing software needed (e.g., Word, PowerPoint).

Request administrative support personnel (typists with selected software experience if at all possible), as they are especially helpful when preparing the final pieces of documentation for the final report.

In cases where the review involves classified facilities/operations, schedule derivative classifiers to come to the work area and review the draft documentation at the end of the day during the last week of the review.

Request sufficient office space and computers to support, at a minimum, each functional area subgroup. Working in one large room is difficult since many discussions take place among site, review team, and field office groups.

Investigate whether laptops can be brought on-site. This is usually not an option when the review involves classified material. The site may require some form of property accountability pass.

Establish clear formal and informal communications between the facility and the review team as early as possible. Define the "rules of the road." This will improve the effectiveness and efficiency of the ORR/RA. Explain the open, "filters off" nature of the daily team meetings, and state that other briefings can be held with management. While important, the number and the length of these meetings should be minimized to allow the members of the review team to do their job.

Team Issues

The ORR Team Leader and Team Members should be identified as early as possible in the startup/restart process. Identification of the ORR Team Leader prior to or during development of the POA allows his/her participation in the POA review and approval process. Early identification of the ORR team facilitates better preparation by team members and allows more time for document review prior to the start of the ORR.

Emphasize that team members are dedicated for the duration of the review. If a team member completes their review before the end of the ORR/RA, and is authorized to leave, their completed Forms 1 and 2 must be signed-off (approved) by the Team Leader before they leave the site. If a summary is due from the team member, the Team Leader must accept the summary prior to the team member leaving the site.

Use site personnel as members of the review team but know that demands for his/her attendance to normally assigned duties can frequently interfere. This can be mitigated somewhat by insisting that site personnel who are team members work in the space allocated to the team for the duration of the review, instead of in their separate office/facilities.

Use a dedicated group of team members to evaluate drills, evolutions, and upsets if possible as it can be valuable to the team leader. A group leader can assign specific observational roles to each group member. Having the group hold a post-event critique will allow development of a form 1 that provides an integrated discussion of the events. The other group members can include discussions of pertinent aspects of the event in their form 1 as required by their assigned CRAD.

Emphasize to team members the need to familiarize themselves with the Implementation Plan and the facility/process under review before the ORR/RA. This should be accomplished through the team members' qualification process, which should establish the minimum requirements for facility familiarity, prior to the review. Additionally, the Team Leader is responsible in the team selection to ensure that the team members have the required technical expertise (education/experience) to perform a valid review of the functional area assigned. This should also be documented through the selection and acceptance process.

Ensure that team members fully understand the scope of the ORR/RA and the time frame in which it must occur. In some cases, time management decisions and sampling plans will be important to successful completion of the ORR/RA within the required schedule.

Ensure counterparts have been assigned to each team member or functional area group. Early identification (usually during the pre-visit) of documents needed, interviews expected, evolutions and drills requested is helpful to both the team member and the counterpart.

Inform counterparts they should be prepared to present information requested. Inform team members they need to be respectful of their counterpart's time constraints and therefore schedule as few after-hours meetings or interviews as possible.

Capture issues and observations in writing as soon as possible. Deficiency Forms (Form 2) should be prepared shortly after an issue is identified. Assessment Forms (Form 1) should be prepared shortly thereafter to capture the full extent of the finding. This will enhance the flow of communication throughout the review and contribute to openness on the part of both the team and the facility.

Developing the Implementation Plan

Ensure the objectives and criteria clearly encompass all aspects of the Core Requirements defined in the approved Plan of Action. Generally, this will be all core objectives in an ORR and some subset of the total for an RA. The ORR standard breaks the eighteen Core Requirements into 36 objectives, which translate easily into functional areas. Experience has shown this methodology to be successful. Additionally, the "geographic" scope of the ORR/RA must be considered. The POA should specify the bounds of the facility, systems, and personnel involved in the activity under review and the approaches for the objectives and criteria should reflect these boundaries.

Confirm that the CRADs are written with a clear understanding of the facility systems and processes under review. A generic CRAD should be tailored during the pre-visit to successfully complete a thorough and critical review. Without this understanding and modification of the CRADs, the review effort will suffer.

ORR team members need to carefully review their CRADs and the CRADs of other team members and identify areas of potential overlap. Areas of potential overlap should be discussed and resolution reached as to who will investigate and document results in that area. This minimizes duplication of effort on part of the team members and the facility, and minimizes the likelihood of conflicting conclusions in the report.

If there are significant delays in commencing an ORR, the Implementation Plan's CRADs should be reviewed again for adequacy. Specific items that may change include: references, scope, programs added or deleted, and member assignments.

Given the level of effort required to evaluate software control systems, consideration should be given to preparing a separate CRAD to address software quality assurance.

Fully involve the team early in the process for training and Implementation Plan development. Early dialogue with facility personnel to gain understanding of the activity in progress and the contractor activities in progress is particularly useful. The Implementation Plan used by the contractor and DOE should be parallel in numerous respects.

Give the POA and the Implementation Plan to oversight groups (EH, DNFSB, State Agencies as required) as soon as possible. Early review and input from all stakeholders will reduce last minute perturbations.

Determine the contract requirements for the facility in reference to Standards Requirements

Information Documents (S/RIDs), Work Smart Standards, DOE Orders, Policies, Manuals, Notices, etc., and Integrated Safety Management. Conduct the review in a manner consistent with the requirements contractually invoked on the facility. Statements regarding the status of the facility in these areas should be included in the final report in addition to the overall evaluation of the compliance posture required.

Defining the Scope and Graded Approach

Define the scope of the DOE ORR consistent with that of the contractor ORR. Differences between the two result in difficulties in performing the DOE ORR. DOE O 425.1C provides a structure for such consistency. Differences will occur in the evaluation of the Core Requirements specifically related to DOE management, which are necessarily absent from the contractor review.

Going beyond the defined scope in a case where an ongoing program that supports the process or facility under review but which is outside the scope of the review, presents potential problems. If the team is to make a judgment that operations can be conducted safely and it requires investigation at the interface of the process/facility with the supporting program, then going beyond the defined scope is proper.

Define the graded approach as it applies to each CRAD as specifically as possible.

Post-ORR Corrective Actions

Although the ORR/RA follow-up corrective actions are not the responsibility of the Team Leader, line management may ask for input and advice as to the necessary action(s) to correct the issues identified during the ORR/RA. The value and effectiveness of the ORR/RA can be significantly decreased by ineffective corrective actions to resolve the issues identified during the ORR/RA. The Team Leader, through interest in the effectiveness of the overall ORR/RA process, retains an interest in the post- ORR/RA corrective actions. In general, the following items need attention:

Facility management sometimes starts taking immediate action to correct symptoms identified by the ORR/RA team without adequate attention to understanding root cause and programmatic basis of issues. As a result, the corrective action may be incomplete, shallow, and short-lived.

DOE line management review of the status of the contractor ORR/RA corrective actions should require corrective action closure packages for the contractor ORR/RA that meet requirements of DOE-STD-3006-2000, section 5.7.3. In that way, the contractor may more effectively evaluate their own corrective actions before reporting readiness to proceed to DOE.

When DOE line management forwards the DOE ORR/RA to the responsible contractor, the actions expected as to the contractor's corrective actions should be clearly stated. Unclear or unstated expectations result in false starts and failure of the responsible contractor to meet DOE's expectations. This in turn has resulted in delays and incomplete actions following several ORR/RAs.

In those cases where issues have been identified as the responsibility of a DOE field organization, DOE-HQ should formally communicate corrective action expectations to the responsible field organization. The expectations should include corrective action plans and closure packages.

Miscellaneous Lessons Learned

Engage a technical editor, if at all possible, during the second week of the review. The final report left at the closeout briefing, although a draft, is the site or facility's first encounter with the full results of the review and as such warrants a technical editing. This person should stay until all Forms 1, Forms 2, and summaries have been approved, the executive summary edited, and the final report formatted, collated, and printed. The introductory material (cover sheet, background, scope, etc.) can be written before the review is started, thus reducing the workload during the review and allowing the team to concentrate on issues at hand.

Do not assign any CRADs to the Team Leader, however technically qualified.

Identify an Assistant Team Leader or Senior Advisor. Having an Assistant Team Leader/Senior

Advisor can greatly improve efficiency in the conduct of the ORR/RA. An Assistant Team

Leader/Senior Advisor is particularly beneficial during the labor intensive ORR preparation phase.

If possible, include at least one trainee on the ORR team so that the pool of experienced readiness review team members continues to increase.

Select a Review Coordinator with previous experience in the ORR/RA process, as this function is essential in the assimilation of the data provided by the team members. Without accuracy and timeliness in this function, conclusions are subject to question.

Select the Team Leader carefully. The Team Leader should be senior to the team members. He or she should not be a peer or a co-worker of the team members. Selection of the appropriate Team Leader is important in that he/she must exercise some degree of control over the team.

Evaluate on-the-job-training, drills, and other training evaluation against the established guidance in DOE Orders, standards, guides, and handbooks. This helps eliminate the subjective review aspects. This should be communicated to the contractor for a contractor ORR/RA.

Generic Lessons Learned That Apply to the Design, Construction, Operation and Decommissioning of DOE Facilities

During startup of new facilities, numerous cases have been observed in which the surveillance requirement supporting the safety basis could not be completed, or if completed, did not properly verify the safety basis requirement. Surveillance procedures should be completed, validated, and executed as part of the startup program, prior to the implementation of the supported Technical Safety Requirements (TSRs). Actually executing the surveillance will reveal such problems as: test points that are not installed; test points that are inaccessible; and other interfering interlocks or functions. These problems have occurred repeatedly in previous ORR/RAs.

Additionally, determination must be made that the surveillance actually tests the function or protective action upon which the safety basis depends. Previous ORR/RAs have revealed numerous occasions in which surveillances have been conducted and through either invalid acceptance criteria or a misunderstanding of the as-built configuration, the surveillance did not test the required functions.