Another year, and another Disaster Recovery (DR) drill is done. We did what we had planned to do in the exercise, and everything largely went according to plan. This has become somewhat of a routine. There were several hiccups, but we did expect little hiccups to happen. Our plan, indeed, include expecting some unexpected variances to happen.
I’ve been involved in DR drills for many years. Some things that never change since the beginning of time is the attitudes people have toward such IT exercises.
There are many stakeholders in such exercises. Higher management likes to see such drills conducted to be reassured that their IT organisation is prepared. This is all part of emergency preparedness.
Customers (i.e. the “users”), on the other hand, are completely disinterested in such exercises. Either they don’t pay attention to them, or they feel whatever else they have to do has far greater importance than such exercises.
Let me give some examples. One of the strangest objections I’ve received to our DR drills was about someone’s lunch with parents of our incoming students. Lunch. DR drill. How are they related? Yes, I understand the DR drill would cause some IT services to be unavailable. But, excuse me, it’s lunch right? You eat during lunch. Why do you need IT services to be available for lunch? How idiotic can that be.
Other stupid reasons are like people who say they need to access email. Yeah, you just absolutely have to schedule your email activity at the same time as the drill? Mind you, we are already staying clear out of work hours.
Or, they have a paper due. Or some sort of deadline. Fine. A deadline means you submit or get the thing done before that date and time. It is not that you must get it done at precisely that date and time eh.
Very often, you could plan a DR drill way in advance, announce it multiple times, and someone comes in at the last week and say they didn’t know, was unaware, forgot, or make up whatever reason, and say they have something else even more important. (Like lunch.) They actually expect you to abandon your far-ahead planning so that their other poorly planned matter takes precedence.
It’s not to say IT cannot give way. After all, we have to work together for the common good of the organisation. Sometimes, we have to compromise and indeed we do. My beef is that some people treat IT like a nuisance child.
Fire drills are equally in bad shape. We conduct fire drills for every building in my workplace just one time a year. It only takes half hour. It is actually quite a big challenge to do just that one drill per year. Building occupants come up with stupid excuses like they will be having a conference call. Or worse, that they will be running some batch processing jobs and may need to receive email and phone queries from users.
There are very good reasons for drills to take precedence over most business operations. It is the perfect opportunity to test how the business handles itself during an emergency incident. This is when business units can re-evaluate their MAO (maximum allowable outage) and RTO (recovery time objective) for various services they depend on.
In the armed forces, one word from the top to do an exercise, and everyone and everything below fall into position to make it happen. I remember how even no-duff (i.e. real, live, and not practice) incidents are handled along-side exercises. It’s an excellent opportunity to test operating procedures, people’s understanding of the procedures, their ability of work under stress, the interfacing and communications between multiple parties, etc.
We’re not an armed forces sort of organisation. We’re at the other extreme, where our users seem bent on blocking the drills. Everyone (or at least the majority) wants to work in a peaceful fairytale dreamland uninterrupted by external nuisances. DR drills and fire drills are fine, just so long as they are not involved, affected or impacted by them.