Business Tech: Disaster Planning

Looking at the news out of Japan reminds us that "wildly unlikely" is not the same as "impossible." Nearly every business above the mom-and-pop level has a disaster plan in place. All of them have a core role for records retention and in most companies, IT is the heart and soul of their records management. That alone earns us a place at the table when the plan is developed.

In the Event of Fire

My family once went out to get Chinese food and, I kid you not, got this fortune in one of the cookies: "In case of fire, remain calm, pay your bill, and leave in an orderly manner." If you'd done disaster planning, you will immediately complain that the wait staff won't be able to stay and collect on those bills. Instead, you'll be setting up procedures for moving the point of sale equipment to the parking lot, having paint handy to establish the queue line for payment, and calculating how far behind the fireman you have to be before setting up the process. The example is a joke, but the concepts are real.

They are real because disaster planning is never about thinking one step ahead or dealing with one vector at a time. You have to look at all the steps — in this case: payment, receipt of payment, recording of payment — and all of the vectors — customer's sequence, wait staff sequence, register operator sequence — to assure a coherent whole. Try fitting all of that into a fortune cookie.

Life Begins at…

All plans must acknowledge that life is more important than things. Remember that a life at risk creates more risk. If Bob is still in the building, he is in jeopardy and so are the firemen who try to rescue him. We have to make sure that every phase of the plan is based on ethical and practical priorities.

Given a choice between people and computers, let the data burn.

We Still Haven't Accounted for Godzilla

I had a lot of opportunity to think about disaster planning when I worked in the international rush courier business. We had multiple chances to test our protocol. You see, when you handle other people's sealed packages you take a bomb threat very seriously. Our problem was 'how much is too much.' I actually sat in a disaster planning meeting once and felt compelled to point out that we had no contingency for an attack by Godzilla. Every plan can be improved, but there is a cost associated with each step.

We do have to account for the likely scenarios. OSHA training includes several of those points. For example, don't have people exit a burning building in a direction which takes them past hazardous chemicals. If you think your building doesn't have any, look at the labels on your cleaning supplies. If you aren't the only tenet in your building - or the only building on the block - you need to know the what and where of your neighbors' storage.

Disaster Planning as a Martial Art

The rule in most forms of martial arts is 'don't be where the punch will land.' Likewise, if your data routinely has two homes, there are zero steps needed to rescue the data from a single disaster. So, off site storage is good, remote hot backup is better. The trade is expense.

When I take the Monday file-save home on Wednesday, the Tuesday save on Thursday, and so on, I have low expense and moderate safety. When I use a courier to take a backup every night to a secure facility, expense goes up but so does my chance of avoiding loss of data. Another reason to use somewhere other than an employee's home: Security rules about moving confidential data to an unsafe location.

My apartment is many things, but has never been touted as a secure facility. Data isn't just data. We live in an age of electronic fraud and identity theft. We live in an age where credit card processing requires PCI compliance. There are standards and rules which might be at war with your common sense need to have off-site back up. The rules were not created to make off-site harder, they jut happen to do that in the process of improving data security. As the great Billy Joel once sang: "All your choices make you change your mind."

Half a Loaf

So, what is the IT obligation in disaster recovery? Data is only one part of the puzzle. Think several steps ahead and ask yourself if you have the configuration files, dlls, programs, scripts, and other information needed to re-create your entire computer from scratch. Look at the mass of connected systems and ask yourself how much of the corporate data is out of IT's hands. If we don't back up three key files on Sue Ann's computer, did we just lose a key segment of our data?

So, when we approach disaster recovery coherently, as a whole, we have to look at not recovery, but a complete restart. Let's take a concrete example. We'll call it Joe's Sporting Goods. The corporation, there never was a Joe outside of the fellow marketing came up with, has fifteen stores, a massive online web presence, and a huge call center and central inventory receiving at his headquarters.

This means that we can't just look at disaster planning for the HQ. We have sixteen physical locations, plus the web. So, ask the IT questions which go with the rest of the disaster plan:

1. Do we need a way to establish sales registers at an alternate location in the event of a disaster?
2. Do we know where to send paychecks if the store is closed?
3. Do we have access to replacement registers? Replacement computers? Replacement networking equipment?
4. For the HQ, we need to know if we have an alternate place to receive and ship goods.
5. For the HQ, we need a fallback call center.
6. What about the web site? Can we shut it down when there is no ability to fulfill orders? Is it hosted remotely? Does that remote hosting have a disaster plan in place which protects our data?
7. Don't forget the data question we started with: Which machines need to be backed up, what data resides corporately, divisionally, by location, and so on.
8. What about database licenses and other product clearance. Do you have the disks, ESD (Electronic Software Delivery) sites, the serial numbers, the proof-of-sale information needed to start up the 'B' site?
9. When a disaster happens, can management get a staff schedule? You can't confirm the building is clear if you don't know who was supposed to be there. Likewise, a guest sign-in list.
10. Your business has many specific issues. How do they affect a restart plan?

As a businessman, I also want another phase to disaster planning: What did I lose? You see, Insurance companies don't like to pay to replace things based on guesses. So, while you are looking at saving the data at a remote location, and looking at replicating hardware/configuration/licenses/etc., you also need to know the reverse. What was subtracted by the earthquake, fire, or flood?

So, disaster planning isn't just data preservation. It is business preservation. Since most disaster plans are chopped into departmental areas, it is easy to forget the totality. Records are of little value to a company which is out of business. We want the data so we can restart, in the original location if possible, or anywhere else we have to restart. If I have the data but no database, I have failed. If I have the hardware but can't configure it, I have failed. We need to use our analytical skills with this problem as we do with other business problems. When I put on my business hat and look at any disaster plan, I primarily want to know one thing: Come hell or high water, I'll still be in business.

Featured:

May/Jun 2011