Contingency Planning

This course will teach you about the importance of contingency planning. You will also learn about PowerTrain's contingency plan and the steps involved when the contingency plan is activated.

Background and Scope

What is Contingency Planning?

The nature of unprecedented disruptions can create confusion, and often predisposes an otherwise competent IT staff towards less efficient practices.  In order to maintain a normal level of efficiency, it is important to decrease real-time process engineering by documenting notification and activation guidelines and procedures, recovery guidelines and procedures, and reconstitution guidelines and procedures prior to the occurrence of a disruption.

During the notification/activation phase, appropriate personnel are apprised of current conditions and damage assessment begins.  During the recovery phase, appropriate personnel take a course of action to recover the  Knowledge Portal components a site other than the one that experienced the disruption.  In the final, reconstitution phase, actions are taken to restore IT system processing capabilities to normal operations.

Why do we have a contingency plan?

The plan has been developed for The Knowledge Portal which is classified as a Moderate impact system, in accordance with Federal Information Processing Standards (FIPS) 199.  FIPS 199 provides guidelines on determining potential impact to organizational operations and assets, and individuals through a formula that examines three security objectives: confidentiality, integrity, and availability.  

The procedures in this plan have been developed for a Moderate impact system and are designed to recover the Knowledge Portal within 72 hours.  The replacement or purchase of new equipment, short-term disruptions lasting less than 8 hours, or loss of data at the primary facility or at the user-desktop levels is outside the scope of this plan.

This plan does not apply to the following situations: 

•    Overall recovery and continuity of mission/business operations.  The Business Continuity Plan (BCP) and Continuity of Operations Plan (COOP) address continuity of business operations.  

•    Emergency evacuation of personnel.  The business emergency evacuation procedures are used to address emergency evacuation.

Concept of Operations

System Description

The Knowledge Portal is a web-based application used to deliver online training for a wide range of Federal clients.  The application was developed to comply with e-Learning standards while providing OPM with a comprehensive knowledge portal and delivering a variety of training courseware content.  

An important aspect of the Knowledge Portal is that the instructional content is not confidential and in most cases is freely available to the public. Information collected and maintained includes user’s names and email addresses can be considered personally identifiable information elements. This information is securely encrypted in separate database tables. 

Three Phases

This plan has been developed to recover and reconstitute the Knowledge Portal using a three-phased approach.  The approach ensures that system recovery and reconstitution efforts are performed in a methodical sequence to maximize the effectiveness of the recovery and reconstitution efforts and minimize system outage time due to errors and omissions.  The three system recovery phases consist of activation and notification, recovery, and reconstitution

Activation and Notification

Activation of the ISCP occurs after a disruption, outage, or disaster that may reasonably extend beyond the recovery time established for a system.  The outage event may result in severe damage to the facility that houses the system, severe damage or loss of equipment, or other damage that typically results in long-term loss.  

Once the plan is activated, the information system stakeholders are notified of a possible long-term outage, and a thorough outage assessment is performed for the information system.  Information from the outage assessment is analyzed and may be used to modify recovery procedures specific to the cause of the outage.  

Recovery

The Recovery phase details the activities and procedures for recovery of the affected system.  Activities and procedures are written at a level such that an appropriately skilled technician can recover the system without intimate system knowledge.  This phase includes notification and awareness escalation procedures for communication of recovery status to system stakeholders.

Reconstitution

The Reconstitution phase defines the actions taken to test and validate system capability and functionality at the original or new permanent location.  This phase consists of two major activities: validating data and operational functionality followed by deactivation of the plan.

Validation

During validation, the system is tested and validated as operational prior to returning operation to its normal state.  Validation procedures include functionality or regression testing, concurrent processing, and/or data validation.  The system is declared recovered and operational by upon successful completion of validation testing.

Deactivation

Deactivation includes activities to notify users of system operational status.  This phase also addresses recovery effort documentation, activity log finalization, incorporation of lessons learned into plan updates, and readying resources for any future events.

Roles and Responsibilities

The Team

Hover over or click the "i" on each image to view the role's responsibilities.

Line of Succession

PowerTrain sets forth an order of succession, in coordination with the order set forth by the organization to ensure that decision-making authority for the Knowledge Portal contingency plan is uninterrupted.  In order to preserve the continuity of operations, individuals designated as key personnel have been assigned an individual who can assume the key personnel’s position if the key personnel is not able to perform their duties.  

Alternate key personnel are named in a line of succession and are notified and trained to assume their alternate role, should the need arise.

Refer to Appendix A in the Contingency Plan for a list of roles. The plan is located on PowerTrain's server. Located on Main> OPM_FedRAMP> PowerTrain Security.

Activation and Notification

Activation Criteria and Procedure

The Knowledge Portal ISCP may be activated if one or more of the following criteria are met:

The type of outage indicates The Knowledge Portal will be down for more than 72 hours.

The facility housing The Knowledge Portal is damaged and may not be available within 72 hours.

There is a service disruption to any part of the Knowledge Portal that the CP Director deems an emergency.

Notification Instructions

The first step upon activation of the contingency plan is notification of appropriate mission/business and system support personnel. 

For The Knowledge Portal, the following method and procedure for notifications are used:

  • CP Director notifies the ODSA. 
  • The ODSA notifies the Contingency Plan Coordinator (CPD)
  • The CPD notifies the Hardware and Software teams
  • In the event of site failure, PowerTrain personnel will receive immediate notification by email and text to PDAs. 
  • Telephone, telecoms, and emails are held between PowerTrain and the Baltimore location.

Outage Assessment

Following notification, a thorough outage assessment is necessary to determine the extent of the disruption, any damage, and expected recovery time.  This outage assessment is conducted by the Outage and Damage Assessment Lead.  Assessment results are provided to the Contingency Planning Coordinator to assist in the coordination of the recovery effort.

Recovery

Sequence of Recovery Operations

The following activities occur during recovery of The Knowledge Portal:

  1. Identify recovery location (if not at original location)
  2. Identify required resources to perform recovery procedures
  3. Retrieve backup and system installation media
  4. Recover hardware and operating system within 72 hours (if required)
  5. Recover system from backup and system installation media

Recovery Notices/Awareness

Notifications during recovery include problem escalation to leadership and status awareness to system owners and users.  

Once the contingency plan is implemented, there will be daily conference calls using WebEx, an online conferencing center, with  to provide status updates and share any relevant issues.

Email updates will be sent to The Knowledge Portal personnel  as site recovery continues.

Note

The two key PowerTrain personnel (CPD and ODAL) are PowerTrain officers and consequently no additional leadership escalation is required within the PowerTrain corporate structure.

Reconstitution

Reconstitution Process

During reconstitution, a determination must be made whether the system has undergone significant change and will require reassessment and reauthorization.  The phase consists of two major activities: (1) validating successful reconstitution and (2) deactivation of the plan.  Concurrent processing is the process of running a system at two separate locations concurrently until there is a level of assurance that the recovered system is operating correctly and securely.

The processes involved with reconstitution involve:

  • Data validation testing
  • Functional validation testing
  • Recovery declaration
  • User notification
  • Cleanup
  • Returning backup media
  • Backing up restored systems
  • Event documentation

Refer to the Contingency plan for additional information on each section.

Conclusion

Conclusion

During a disaster situation, the availability of the contingency plan is essential to the success of the restoration efforts.  The Contingency Plan Team has immediate access to the plan upon notification of an emergency.  The Contingency Plan Coordinator ensures that a copy of the most current version of the Contingency Plan is maintained on PowerTrain secure server, Main>OPM_FedRAMP>PowerTrain Security.

Contingency Plan Team members are obligated to inform the PM, if and when, they no longer require a copy of the plan.  In addition, each recipient of the plan is obligated to return or destroy any portion of the plan that is no longer needed and upon termination from PowerTrain.

To complete the training, please answer the questions in the Knowledge Check section.

Knowledge Check

The contingency plan is activated if the Knowledge Portal will be unavailable for more than:

  • 12 hours
  • 24 hours
  • 48 hours
  • 72 hours

The three phases are:

  • Activation and Notification, Recovery, and Rebuilding
  • Activation and Notification, Recovery, and Reconstitution
  • Activation and Notification, Retrieving, and Reconstitution
  • Activation and Notation, Recovery, and Reconstitution

True or False

  • The plan is located on the PowerTrain server