• Overview of Business Resumption Planning

• Justifying and Developing a Business Resumption Plan

Phases in developing a typical BRP
Prudent Person justification methodology and case study

• Conclusion

Overview of Business Resumption Planning

Recent experience in the information systems contingency planning area has demonstrated that disasters to quality business offices/facilities occur approximately once every hundred years. A disaster probability of 1% per year appears to be the proper basis of individual facility based business resumption planning. This is actually a very high probability when the prudent business person realizes that the loss of a business facility containing critical functions can destroy the company. The advent of critical data processing applications, as discussed in Andrews [1], first brought this exposure to the attention of most executives.

Development of the BRP Field

• Support for developing a contingency plan

Several organizations sell an IS contingency plan development methodology including planning procedures and a sample plan available on a diskette. The sample plan is tailored, often with consulting help from the vendor, to the organizations individual requirements. The sample plans are often of great value to organizations who have not had prior experience. However, their use often leads to long wordy plans that are not read by potential emergency operations and recovery participants.

• Shared backup data centers

Two large and numerous smaller organizations offer data center operations facilities containing operating computers (a hot site), fully equipped space for installing several additional computers (a cold site), and space for clerical and support personnel. Scheduled use of the hot site computer(s) is only for testing by the eighty to a hundred clients of the hot/cold site facility. These commercial backup sites are extremely valuable after a localized disaster that makes a data center unavailable, but are subject to prior or multiple occupancy following a wide spread disaster affecting many facilities.

Contents of a Typical BRP

The typical business resumption plan contains three types of information: backup resource arrangements; BRP procedures for notification, activation, mobilization, and emergency operations; and listings of equipment and other resources at all facilities. It requires two very different formats, that is the information needed before and after a disaster.

• BRP information needed before a disaster

A formal BRP plan is needed for orientation of employees that will be involved in activating the plan and performing emergency operations. Orientation material is also needed for all employees in both the life safety and BRP areas. Detailed emergency operations plans are also needed both before a disaster for use during testing, and after a disaster for use during emergency operations. The type of information needed includes:

Backup resource arrangements:  The initial and most critical activity in creating a realistic BRP, is arranging for backup resources and data for use after a disaster.

Data centers with critical online applications either use commercial backup centers or utilize dual data center architectures. IS users and manual processing organizations also need to have backup space and resources available. Most organizations have space and equipment occupied by non-critical processing activities (such as development organizations, conference rooms, exhibit space, executive offices, etc.) that can be utilized in an emergency, given sufficient planning.

The lack of copies of paper based data, forms, and procedures is the most common weakness in BRP arrangements. Data centers routinely send copies of all data files to off-site storage daily. Manual processing organizations must also store copies of all critical data off-site, frequently as microfilm. The loss of desk-top papers, diskettes, and Rolodexes can cripple organizations.

Applicability
Distribution list (Controlled)
Organization structure (Team definitions)
Notification trees (Business and home numbers)
Activation and mobilization procedures
Backup resources (Locations and functions)
Emergency operations policies (For each business functions teams)
Resource recovery policies
Testing and training policies
Appendixes (copies of material for use following a disaster)

These site specific wallet size cards are distributed to all employees. They should contain both life-safety and BRP information such as assembly locations, emergency operations policy, and numbers to call for information.

These tabulations contain such information as: lists of all resources by location including replacement information and backup resource locations; where all personnel are to report during emergency operations; and where to forward data and materials from off-site storage.

The best BRP documentation for use following a disaster that the author has observed, consisted of approximately a dozen reference cards, several books of resource tabulations, and two well equipped EOCs.

The initial step in any BRP program is that of obtaining the substantial funding normally required. This requires convincing the Board of Directors on the reality of a possible disaster and its probable impact on the ability of the organization to survive.

A detailed quantitative approach to a risk analysis is used widely. The approach is popular with government and large decentralized industrial firms with major consulting budgets, see Wong [11]. It determines the probability of various man-made and natural disasters occurring and their impact on key business functions. The results present probability estimates that are difficult to translate into risk-cost-protection decisions. Additionally they are actuarial based and do not apply to business decisions involving a single site or resource.

The board of directors of most firms respond far better to a fiduciary responsibility based risk analysis. A list of risks to which the firm's facilities and personnel is exposed is presented and a case study approach is used to demonstrate realistic risk exposure in Waldman [10]. Estimates are made of the financial impact on various business functions, computer related and non-computer related, of a loss in resource capability. When the impacts include financial or service level losses that can effect the firms survival, then the board members fiduciary responsibility requires a prudent level of protection and recovery capability. Funding for an adequate BRP is then made available, often as a priority project.

Following initial funding, the next step in the BRP program, is to determine the possible extent of exposure to natural and man-made disasters of critical resources; including facilities, data, and personnel. Procedures must then be implemented to minimize the probability of such disasters occurring.

Physical security planning primarily involves access controls, fire and water protection, earthquake and storm hardening, and critical records security. Most firms have a physical security program in place covering these areas prior to the implementation of a BRP program. The next step in the BRP is, therefore, simply an assessment of the program, and improvement if necessary. The author's experience indicates that the critical records area, particularly for non-computerized files, frequently is a major weak point.

Data security and protection programs are not as wide spread as physical security programs. Few firms have high quality data oriented security programs involving off-site backup storage of critical paper based financial & personnel records. Protection in this area frequently requires a major effort.

Disaster planning, as illustrated in Rohde and Haskett [6], is often initiated by the organization's data center, as it first implements applications critical to the day-to-day operations of the organization. The selling of data processing oriented disaster planning to the Board of Directors often alerts them to the risk presented by the non-computerized portions of the firms operations, and a total disaster recovery planning effort is initiated.

A team effort is the best approach to creating an organization's initial BRP. The team should include at least a person experienced in BRP architectures and plan development; a person with long term responsibility for developing and maintaining the plan; and an influential manager with in-depth knowledge of the organization, its operations, and its people. The team should move through the development cycle backwards and then forward.

Step 1 - for the various major resources of each business unit, determine the potential recovery architectures available, their costs, and the recovery periods they offer.

Step 2 - perform a risk analysis determining which resources are truly critical to the organizations survival. For those resources, determine their desired recovery periods and the most practical recovery architecture approach for each.

Step 3 - present a business resumption policy to the Board of Directors balancing risk, costs, and service levels.

Step 4 - create a detailed design of the authorized recovery architectures and assist each business unit in creating a business resumption policy and architecture.

Step 5 - assist each business unit in designating a BRP coordinator,   assigning a planning team, and assisting them in developing and testing their plan.

Desk-top walk through - Before any detailed testing, key stakeholders in each business function's BRP are convened in a conference room, and a detailed review is performed of the plan. Many small events are described and the participants are asked to state how the plan would guide their reactions. The events should require utilization of: major backup resources, emergency operations approaches, and all emergency response teams. Following this step, operations and simulation tests are scheduled.

Operational testing - Few organizations operationally test the complete disaster reaction cycle of: activation, life-safety, damage assessment, mobilization, emergency operations using off-site files and backup resources, and recovery planning. Only the data processing emergency operations area can normally be tested without involving a substantial number of persons during business hours. The scope of most operational tests therefore, includes: a semi-annual off-hour call to the manager of data center operations, assembly of the backup site operations team, acquisition of backup materials from an off-site location, travel to a backup hot/cold site, installation of systems and applications software, loading production data, and systems test of several critical applications.

Simulation testing - Simulation is the most feasible approach for testing the decision making aspects of disaster reaction activities, see Rosenthal [8 & 9]. The use of simulation exercises for testing a BRP has been spreading slowly over the last decade. Unlike their military counterpart, war games that use computer driven scenarios to perform very realistic exercises, BRP exercises are paper and pencil simulations. Teams are placed at tables representing their backup locations, and the description of an evolving disaster is presented. The teams communicate using backup communication resources or forms, make decisions, and everyone pretends that what is ordered actually happens. Debriefings and evaluation studies follow to correct any flaws in the BRP.

A scenario for use in simulation testing of a BRP must fulfill several objectives.

Except under very unusual political conditions, a major failure during a simulation test is not a suitable motivator for improvement in disaster planning, or for acquiring additional funding for backup resources. A primarily positive experience however, appears to be a powerful motivator to obtain additional funding to complete planning and backup resource acquisition. In fact, the scheduling of a simulation is often the easiest way to motivate organizations to update their staffing and contact lists.

Proper planning of a scenario includes the review of each participating organization's BRP to determine if they can perform emergency operations at an acceptable level. If they cannot, a discussion with top management is appropriate, and warnings to the deficient organization's management is always proper (e.g., there should be no major surprises or disappointments).

Simulation exercises with four to six time steps are the most practical. Each time step must meet the following criteria:

a) The external and internal environment should change in terms of both the evolution of the events causing the disaster, and in terms of emergency and recovery efforts (e.g., new information given to participants and new actions required).

b) Each team should have some significant action they must accomplish (e.g., a decision, announcement, report to management).

c) Time allowed for the time step should be sufficient but not generous (normally 30-60 minutes).

The period simulated becomes longer with each time step in the simulation. The simulations performed to date indicate that the initial period simulated is often one to four hours, while the final period simulated is often several days to a week.

Messages and plans produced during the simulation exercise should be rigidly formatted and documented, so that there is a detailed record of all events and actions. This documentation, together with the umpires and evaluators check lists, is necessary for later analysis.

Most simulation exercises are very successful in that they force personnel to learn the BRP while working together, and find flaws and inconsistencies in emergency operations and recovery policies and plans.

There is a substantial methodological and financial difference between data center and manual business functions risk management decisions using 'prudent business' and traditional 'probability' based methodologies. There is also a vast difference between risk management of a data center's processing of the record keeping applications of the 1960's through the early 1980's, and the risk management of the critical on-line operational applications of the late 1980's through the 1990's. The hypothesis of Waldman’s thesis [10]. is that the probability based approach, created for the record keeping applications of the early years of Information Systems (IS), is no longer appropriate for the mission critical applications of the 1990's, and has never been appropriate for critical manual systems.

• The prudent person methodology is based on executives eliminating those alternatives that risk the short and long term viability of the firm, and analysts then selecting the lowest cost alternative that provides acceptable recovery times. The prudent person duty of care test requires that an officer make a "reasonable investigation and honestly believe that their decision is in the best interest of the corporation", see Metzger et al. [4].

• The probability based approach is based on analysts multiplying the potential loss experienced following a disaster by the probability of it occurring, and comparing that to the cost of backup alternatives. As an example, the IBM approach as defined in Wong [11], states that "For each system ... the expected frequency of occurrence per annum, P, as well as the loss incurred, V, are calculated ...and the exposure, E, per annum is then evaluated from the values of P and C". This approach often exposes the organization to unacceptable losses when a low probability disaster actually occurs.

A comparison of these approaches is illustrated in the simulated conversation that follows. It is extracted from a recent speech by the authors to the San Fernando Valley Association of Contingency Planners. The illustration involves a simulated discussion between an executive, an insurance oriented financial analyst ( Mr. Probability) and a management systems oriented MIS planner (Mr. Application). 

The discussion is as follows.

Mr. Executive Speaks

"Mr. Probability, I understand you do not agree with Mr. Application’s request for a budget increase of $500,000 per year to implement a contingency planning program for our computer and data communication systems, as well as for semi-annual testing using a commercially available data center back-up site. Why do you feel the expenditures are financially unjustified?"

Mr. Probability Speaks

"Mr. Executive, I have contacted the proposed data center backup site vendor and requested his experience over the last decade, concerning how often a data center suffers a disaster. They have found that each of their backup centers with subscription levels of approximately 100, are used approximately once a year for other than reactive testing or planned conversions. I am therefore stating that a data center disaster is a once in a hundred year occurrence."

"Secondly, Mr. Applications request indicates that the old data center, that was replaced ten years ago because of security and size consideration, is still functional as a backup site capable of processing our production operations within a week following emergency ordering of equipment.

As he states, it can be populated with computers etc. within a week and the CIS department could have our production systems current in two weeks. The backup data center he wishes to subscribe too, plus our testing expenses, will cost $50,000 per month and would permit him to have our productions systems current in two days. We are therefore computing that $500,000 per year is worth 10 days of operational losses.

"Thirdly, Mr. Applications indicates that direct losses due to non-current production systems are $3,000,000 per day. This figure was arrived at by accounting based on lost contributions to overhead and profit of three days sales.

"Lastly, if we multiply the $3,000,000 by 10 days we get $30,000,000 additional potential loss in a disaster if we do not subscribe for the commercial backup center. However when we divide this figure by 100, the probability of loss per year, we get a value of the commercial backup center of $300,000 per year. This net loss of $200,000 per year, is obviously a bad investment of capital."

Mr. Executive Speaks

"Mr. Application, if your figures show such a bad investment, why have you proposed the contract?"

Mr. Application Speaks

"Mr. Probability does not understand the implications of ten days down time, now that our production applications are online. It is no longer a question of lost sales but of retaining customers’ business after the ten day outage. I have had extensive conversations with marketing and customer service management. They feel that half our customers can tolerate a 10 day wait and will continue to buy from us. However marketing believes it will take several years to win back the other half of our current customers. Their optimistic figure of an average of a year to resell a customer will give us an additional loss of $18,000,000 ($3,000,000 times one-half times 240 days). This will give us a total loss of $48,000,000. If we then divide by 100 we get a return of $480,000 per year on an investment of $500,000. E.g. a break-even situation.

"However, this type argument, including the division by 100, is irrelevant. The problem is that the data center could be destroyed tomorrow, or next year or not for 200 years. If it is destroyed and we did not take prudent steps to safeguard this critical business resource, we will all lose our jobs and be subject to stockholder suites.

"The loss of approximately $48,000,000 is almost half of our annual profits.

Additionally the loss of half our customers and the reduced service level for the remainder will ruin our reputation for service that is our future. We may never recover, let alone gain back the business we lost in a year on average.

"I therefore recommend that, as a prudent business executive, you must protect such a critical resource as our data center and give us the $500,000 per year additional budget which is an addition of 2% to our total budget."

This simulated discussion illustrates how subtle aspects in the presentation of information can significantly impact decision making-particularly when uncertainty is involved. Throughout the literature on decision making, there is substantial evidence to suggest that intuitions about risk routinely deviate from rationality, because executives do not typically appreciate the nature of uncertainty. This simulated discussion illustrates that the decision on how much safety is worth, is very difficult.

Risk Analysis

This example is based on a consulting study, performed during in the early 1990's, of a wholesale distributor’s data center contingency planning project. The planning project determined that the proposed security and data protection plan would leave their data center vulnerable to natural and man made disasters, such as: fire, earthquake, utility interruption, and strikes. The objective of the planning study was to recommend an architecture that provided for continued processing of their critical applications. These included order processing, inventory management, accounts receivable, and payroll.

Alternative Scenarios

Four alternative BRP architecture scenarios were available. These were: a dual center approach, use of a vendor hot/cold site, use of a current company facility as a cold site, and continuation of their current approach with no backup resources.

• The dual center approach involves the construction of a new secure data center facility, as well as the splitting of processing between the current and the new center. The typical approach for this type of architecture is discussed in detail in the article by Rosenthal [7], entitled "The Emerging Enterprise System Architecture". The expected maximum down time after a disaster using this approach is several hours.

• The vendor hot/cold site approach involves subscribing to a commercial recovery center with compatible mainframe systems. The expected maximum down time after a disaster is several days.
  

• The cold site approach involves the equipping of a current warehouse facility with all environmental and communication facilities required to quickly install a duplicate of their current data center. The expected maximum down time after a disaster is several weeks.
  

• A continuation of their current no backup resource approach will lead to a expected maximum down time, after a disaster, of several months.

Forecasted Annual Expenses

Table 1: Annual Scenario Expenses, presents the then estimated annual expenses of each scenario. The total annual information technology budget for the organization approximates $12,000,000. The dual data alternative with down-time of several hours represents approximately 6%, and the vendor hot/cold site alternative with down-time of several days represents approximately 2%of the annual IT budget.

(1) 7 year amortization
(2) 5 year amortization

Step 1: Forecasted Annual Losses

The following analysis presents both the losses incurred by the organization during recovery of normal information technology processing, the losses incurred in winning back any lost customers, and reestablishing their service level reputation.

Step 2: Estimated Order Retention Rates

Figure 1: Projected Order Retention Rates, were derived from interviews with key marketing and management personnel. As a wholesaler, the organization believes their customers would switch to alternative suppliers for new orders within four days. This would be caused by the lack of inventory information and the delays in picking and shipping cause by reversion to a slow manual operation and a shortage of trained personnel. They also estimate that reorders of proprietary items, representing 65% of reorders, would continue, but reorders of generally available items would stop over a three week period. Figure 1, is derived in the spreadsheet included in Appendix A of Waldman [10].

Step 3: Estimated Order Rates

Figure 2: Projected Order Recovery Rate After A Disaster, presents the estimated rates of recovery of orders following an interruption due to lack of IT capability after a disaster. The key marketing and management personnel believe the firm could recover approximately 5% of their former order rate per month after return of full IT capability. However, recovery from the vendor hot/cold scenario is slower, since all lost orders were new orders. Figure 2, is also derived in the spreadsheet included in Appendix A of Waldman [10].

Step 4: Forecast Economic Impacts

Table 2: Economic Analysis of Scenarios, computes the impact of the scenarios from both the

The use of the following recommended fiduciary responsibility approach is illustrated for each of the defined scenarios.

• Months after Data Center Down-Time

• Order Percentage

• Average Order Retention Percentage

• Hot Site Recovery

• Cold Site Recovery

• No Backup Recovery

Figure 2

Projected Order Recovery Rates

This approach would lead to an approximated loss of $30,000,000. This represents a loss of 70% of annual profits, which would be a disastrous impact on the company. The board would immediately have to sell the firm, or cease operations. This level of loss was totally unacceptable.

Probability Analysis Approach

The data for a probability analysis of the distributor’s data center contingency planning is included in Table 2.

Contact the Author:  ~  Click Here