Disaster Recovery Drill

With the push of this button, the air circuit breaker is switched off, and the disaster recovery (DR) drill is officially started. Yes, if you notice from the photo, the circuit breaker is huge. It is not like any common circuit breaker you find in your household electrical panel. This is a huge circuit breaker sized for 2500 amperes 3-phase current. It’s 415V, more than the typical 230V household power supply, but in the grander scheme of things, this is still considered “low tension”.

DR drills are common exercises carried out in many organizations for variety of purposes. For us, this drill is for a data centre. The plan is for this to be a yearly exercise. It is an opportunity to test various contingency scenarios. It’s not just to test the redundant/failover systems, but also to ensure that operational staff are familiar with the response plan.

Here’s a photo of the data centre when the drill is in operation. The room has somewhat “romantic” lighting. It’s from the abundance of emergency halogen lights. It was our intent to provide so many emergency lights so that the room would be sufficiently bright in order to actually be able to work in it during a blackout. In one of our previous sites, some of the lights are actually powered off the UPS. It’s something we’ve leant from experience that lighting is pretty important.

Our drills are always very eventful. Something or other will screw up royally. The only good thing is that in recent drills, the exercise for the most part has managed to keep to the prescribed timetable. Touch wood.

Yesterday’s drill was just like that. The drill was largely completed on time, in the sense that we could “dismiss” the main body of people. But there was a real gigantic disaster in the middle of the drill. It wasn’t planned to be part of the drill. Nevertheless, it also served as a good real-life test of our response to the unexpected incident.

One of my favourite stories about business continuity and disaster recovery planning is that of Dow Jones in the aftermath of the 9-11 terrorist attack of the World Trade Centre in New York. Dow Jones’ office was not at the WTC, but it was just opposite it. Their office was required to be relocated. Their business continuity and disaster recovery plan was so effective that they could resume business and continue to provide its readers with their Wall Street Journal the very next day. Just like that.