News just in from Tech In Asia, a leaked internal incident report from IDA has revealed that their ISP “routing problems”, coupled with the failure of an Internet router, caused the massive outage last Saturday that took out 19 government websites.
The incident report stated that Internet access to GDC1 (which I’ll hazard a guess as referring to the “Government Data Centre 1”) was disrupted from 3pm to 5:21pm on Saturday 2nd November.
The report went on to itemise the impact of the outage to members of the public:
- No access to the Government Internet systems and e-services hosted at GDC1.
- Unavailability of the Government DNS service.
Now, that wasn’t so difficult to say, was it? I’m sure this would have been a whole lot more satisfying than IDA’s statement, after the disruption had commenced, that the government websites were under planned maintenance.
So basically, we are hearing from the leaked incident report now that the outage was due to Internet accessibility. at GDC1. How interesting.
The maintenance was supposedly on the Internet access. The outage was about the Internet access. I wonder, how did that get translated into “website maintenance”.
Incidentally, during the period of DNS outage on Saturday, access to the IDA website (www.ida.gov.sg) was actually alright, if you happen to know its IP address, or had the benefit of using a DNS server that still held a cache entry for its IP address. Sure, it is possible that somehow, the IDA website was in a different data centre from GDC1.
Well, for now, let’s just say this incident report is a whole lot more satisfactory that the “planned website maintenance” originally posted. I’ve new questions to ask now:
- Was this maintenance to implement a security solution something carefully planned a long time ago? Or was it something pushed through at the 11th hour because of the threats from Anonymous?
- We are talking about the Government Data Centre here. Why was it not discovered sooner that the secondary Internet link was not functioning properly?
- I understand that with two Internet links, you could shutdown and do maintenance on one link while active traffic is routed to the other. Even then, this seems a little risky to be done during day time hours?
This smells of a rush job to me.
See my previous post: Singapore Government Under Attack
Note: The Cisco 2600 router picture above is a stock photo from my image library. It has nothing to do with this incident.