How I Crashed My Website

Well, my fingers were itchy, and I wanted to test something out. Something that carried a small but inherent risk of mucking up. I was careful, no less, to make sure there would be no mistakes, no typos. In fact, I wrote a “rollback” script to undo the changes after a timed delay, just as a precaution. So, even if the changes mucked up the system, the changes would be automatically rolled back anyway. Then, how about testing this rollback script too? Ah, that’s where it screwed up. Something else was broken, and the rollback script got stuck, and it disconnected my server from the net.

And so in the dead of the night, I had to make a trip down to the colo data centre to fix my server. The security guard was delighted to have some company. I wasn’t. Well at least not in the wee hours of a Monday morning. He was in the mood to chat, and I tried my best to be polite.

The best way to keep a server running “forever” is to not touch it at all. Itchy fingers crash servers. Reboot a server, and maybe it never comes back up. (Oh yes, I have seen that many times.) So far in my experience, about the only things that spontaneously break down on their own are hard disks.

Just for the interest of those who have not seen real servers, the above picture shows where this website actually lives. It is a rackmount server measuring about 17″ wide, 1.75″ high. This is what the industry refers to as a “1U” rackmount server. The “U” is a measurement of “rack unit”, 1,75″ high, corresponding to standard dimensions of equipment that these equipment racks are designed to accommodate. Equipment depths can vary widely, but rarely exceed 30″ deep. Although I run data centres too, I pay to rent space in someone else’s commercial data centre to host this website.

I was looking around this colo data centre and observed how many other customers still don’t install proper rackmount servers. The most common “servers” are actually PCs contained in desktop or tower chassis (casing). In fact, many “well known” Singapore websites are hosted in this place using clunky tower PCs. You could maybe house 4 or 6 of these tower PC boxes in a standard equipment rack. If you had a proper 1U rackmount server (i.e. like the one pictured above), you could fit 42 of them in a standard equipment rack. It would save you tremendous amount of space.

Data centre space is extremely costly. It costs anywhere from S$5K to S$10K per square metre to turn an office space into a data centre. Yes, that excludes the cost of building up the room (i.e. the building structure such as the floor, walls and ceiling). As a result, rental of rack space can be quite costly. This was the case in the dot-com boom.

Rental costs have dropped since the dot-com bubble burst, of course. But the real cost of building a data centre remains high.

Anyway, here’s how a blade server looks like. As data centre space is precious, there is this continual drive to shrink servers. Pictured here are 2 blade chassis, accommodating a total of 28 servers. Total height is only 24.5″ (or about 63cm). Mind you, these are very high-end powerful servers. Not some miniaturized classic Pentium PC. How’s that for space efficiency?