With the push of this button, the air circuit breaker is switched off, and the disaster recovery (DR) drill is officially started. Yes, if you notice from the photo, the circuit breaker is huge. It is not like any common circuit breaker you find in your household electrical panel. This is a huge circuit breaker sized for 2500 amperes 3-phase current. It’s 415V, more than the typical 230V household power supply, but in the grander scheme of things, this is still considered “low tension”.

DR drills are common exercises carried out in many organizations for variety of purposes. For us, this drill is for a data centre. The plan is for this to be a yearly exercise. It is an opportunity to test various contingency scenarios. It’s not just to test the redundant/failover systems, but also to ensure that operational staff are familiar with the response plan.

Here’s a photo of the data centre when the drill is in operation. The room has somewhat “romantic” lighting. It’s from the abundance of emergency halogen lights. It was our intent to provide so many emergency lights so that the room would be sufficiently bright in order to actually be able to work in it during a blackout. In one of our previous sites, some of the lights are actually powered off the UPS. It’s something we’ve leant from experience that lighting is pretty important.

Our drills are always very eventful. Something or other will screw up royally. The only good thing is that in recent drills, the exercise for the most part has managed to keep to the prescribed timetable. Touch wood.

Yesterday’s drill was just like that. The drill was largely completed on time, in the sense that we could “dismiss” the main body of people. But there was a real gigantic disaster in the middle of the drill. It wasn’t planned to be part of the drill. Nevertheless, it also served as a good real-life test of our response to the unexpected incident.

One of my favourite stories about business continuity and disaster recovery planning is that of Dow Jones in the aftermath of the 9-11 terrorist attack of the World Trade Centre in New York. Dow Jones’ office was not at the WTC, but it was just opposite it. Their office was required to be relocated. Their business continuity and disaster recovery plan was so effective that they could resume business and continue to provide its readers with their Wall Street Journal the very next day. Just like that.

Related posts:

1. Hot Data Centre, Very Very Hot Data centres pack lots of cool technology in them. The...
2. How I Crashed My Website Well, my fingers were itchy, and I wanted to test...
3. Fire Suppression Gas Discharged We witnessed the discharge of HFC227ea (FM200) fire suppression...

Last weekend, zitseng.com and the rest of my servers were moved to a new data centre. The rental costs at the old place had sky-rocketed and it was not economical for us to stay with them anymore. This was responsible for some of the down time of this site, and partly the reason for the quietness or lack of posts recently.

Moving servers is like moving house. You have to disassemble, pack, move, unpack and reassemble. I have moved many servers before, but the complexity of this 5-server move actually surprised me. Although we marked out a maintenance window from 2pm to 10pm, we imagined that we would be done by about 5pm or 6pm at the latest. But in the end, it took us up to 9pm, and technically there was still a bunch of loose ends that we did not resolve until the next day. Probably a key difference between this move and the others I’ve done is that in the other cases, I run the network and facilities at both ends.

The days of cheap data centre co-location packages are coming to an end. Many data centres have finally realized that they’ve gotten their sums wrong. I run data centres myself in my full-time job, and I know the utility consumption costs alone (not including capital expenditure, maintenance costs, manpower costs and other operational costs) can exceed the per-month price charged by some commercial data centres. So many data centres are now re-pricing their co-location charges upwards, sometimes as much as quadruple the original charges.

This new place seems like a simple setup. Everything was running fine until the 3rd day (technically, it was not even 48 hours from the time we setup), when we were plagued with a 7+ hours network outage. Rather dissappointing that the problem took so long to get resolved, but I’m hoping that this is an one-off incident that will not repeat.

So this network outage took this site offline again. In the 4 years or so in our previous data centre, there was only one incident due to a blackout that lasted several hours too. Yah, I know, data centres are supposed to have UPS and generator backups. But accidents still happen. That incident caused us hardware damage, so it was actually more disruptive and costly to us.

While checking up things today… hey, WordPress version 2.6.2 has been released. So okay, this site is updated, particularly since it fixes some security issue (though it seems a little obscure). It’s always best to keep abreast with security updates. It may be a problem with a feature you’re not using, so perhaps it is not urgent, but what if some day you decide to use that feature.

So that’s it on the recent developments here at zitseng.com.

Related posts:

1. Hot Data Centre, Very Very Hot Data centres pack lots of cool technology in them. The...
2. Water-Cooling the Modern Data Centre We have a data centre that recently became the first...
3. The M1 3G network is down Seems it is now M1’s turn to have problems with...

We’re starting to recruit student helpers again. They will help us with our network and data centre operations. This is like an internship, a program that we’ve run for one semester already. It is a really exciting opportunity for students as they will get hands-on learning experience working with production enterprise networks and data centres.

The students in this programme are called “Student Helpers”, although at some point we’d like to rename or promote this to a better sounding term like “Student Associates”. This isn’t the first programme that we’re running, but it is the first that seems to be working out. It’s a good thing, because sometimes what we find terribly exciting doesn’t quite match up to what students think or want.

What do our student network engineers do? Basically, they help us run our production networks and data centres. They will get to learn many things that are not taught in books, touch really cool high-end stuffs, and spice up their resumes.

These are real enterprise-scale operations, bigger than what you’d see in typical SMEs. We do many things that are “cutting-edge”. Such opportunities are usually not even available to fresh graduates, let alone undergraduate students. So this is really something quite exciting and valuable. Well, at least that’s how we hope our students will see it.

Of course the next question then is, what is there in it for us? Well, we have too many things on our hands, too little time, and not enough people. We could use with some help. Students are, admittedly, cheap labour. But we see this as a mutually beneficial relationship, and we hope to get some really good students onboard. It will cost us time and effort to run the programme of course, but hopefully we reap more benefits out of it than the overheads will cost us.

This is not the place for a total newbie to start learning the very basics of networking. So one thing we will tell potential students is that we expect them to come in with either some basic understanding of networking, or a willingness to put in the effort to quickly get up to speed. The right aptitude and great passion are important. After all, we are expecting them to do useful work for us, we aren’t really running a training school here.

I hope we will be just as successful (or more so) with our second batch.

Related posts:

1. We’re Hiring Super Engineers Are you looking around for a new job? Or perhaps...
2. Cisco Career Certifications We just started running the CCNA (Cisco Certified Network Associate)...
3. Student suspended for bypassing network security According to this article from Network World, a student at...

A tiny hairline crack, but it still takes a significant effort to rectify it. Can you spot the crack? That tiny crack led to water leak which is a significant issue in a data centre. The problem erupted two weekends ago when our water leak detection system signaled an alarm. Initially, it looked like it might be a problem with condensation on the chill water pipe. However, after ripping away some insulation to track down the source of the water leak, the hairline crack was discovered.

Fortunately the water leak was contained in a drip pan, and we ran a water leak detection cable in the drip pan to detect any accumulation of water. That’s how we got wind of the problem. I must say water leak detection is really quite important, particularly after hearing so many stories about various facilities having their under floor space flooded with water for a long while before anyone noticed anything was amiss.

The hairline crack happened on the water valve. As you might tell from the photo, the valve is quite a solid piece of metal. It’s quite heavy too. It’s quite unusual for the valve to crack.

It took practically all of 4 hours to rectify the problem: Replacing the valve itself as well as all the ancillary works related to it.

Related posts:

1. Water-Cooling the Modern Data Centre We have a data centre that recently became the first...
2. Vanessa’s Back to Hospital The day before, Vanessa started getting a high fever and...
3. Google Phone Hits Problems Google’s new phone, the Nexus One, has started to turn...

We’re still looking to hire more staff to support our Network, Data Centre and IT Security operations. It is the same super engineers kind of work I blogged previously, but this a technologist position targeted at polytechnic graduates. Our priority for this position is in Network, Data Centre and IT Security operations, in decreasing order. Let me know if you’re interested.

Related posts:

1. We’re Hiring Super Engineers Are you looking around for a new job? Or perhaps...
2. Recruiting Student Network Engineers We’re starting to recruit student helpers again. They will help...
3. SCADA Security Talk This would be my first visit to Nanyang Polytechnic. I...

We witnessed the discharge of HFC227ea (FM200) fire suppression gas a few days ago! It was really a special and unique experience because few people ever get to witness the discharge of FM200 gas.

FM200 is a clean agent residue free fire suppression gas often used in data centres. FM200 gas, instead of water or other chemicals, is discharged into the room to extinguish a fire. The main advantages of FM200 is that it is safe to use around all types of electrical and electronics equipment, safe for humans, and leaves no residue after the gas has been discharged.

Actually, the gas is HFC227ea, and FM200 is the name of the product you buy from Great Lakes. (This is a bit like how Xerox has become synonymous with “photocopier”.) There are other types of fire suppression systems that use HFC227ea gas and they may be marketed under different brand names.

One of the most important benefit of a clean agent residue free fire suppression system in a data centre environment is that IT operations can continue uninterrupted when the gas is discharged. Servers, network equipment, disk storage, and all sorts of equipment just continue running as normal. Furthermore, because there is no residue left behind after discharge, there is no clean-up. In fact, the gas clears away fairly quickly, so people can return to the room very quickly after the fire has been put out. As a result, there is very minimal impact to business operations.

Had water been used, you’d be quite certain all electrical systems would have short-circuited, and thereafter, not only would you be left with damaged equipment, you’d also have a mess in the room to clean up.

HFC227ea gas is not particularly cheap. As a result, no one actually discharges HFC227ea gas as part of a system testing and commissioning exercise. No one discharges the gas for testing or demonstration purposes either. This makes our experience rather special. So how did we get to experience our gas discharge? Well, *secret*.

Related posts:

1. False Fire Alarm Again All the false fire alarms at my home are getting...
2. Fire Alarm is No Alarm [photopress:20070901136.jpg,thumb,pp_image]Today there was a false fire alarm at my condo....
3. How To Make Your Notebook Batteries Last Longer Everyone wants to make their batteries last longer, both in...

Are you looking around for a new job? Or perhaps your first job? Well, we’re hiring! We’re recruiting to fill the vacancy of Systems Engineer (Network). You will work with our Network Team which manages overall Network operations, Data Centre operations and IT Security operations. (Yes, if it sounds like three jobs rolled into one, it is.)

What are the perks of this job? Well, you’ll get lots and lots of learning opportunities in a relaxed and friendly working environment. We do run state-of-the-art network infrastructure and data centre facilities, so it is not like you’re stuck with some yester-year’s technologies.

Why Super Engineers? Well, the term Systems Engineers is often abbreviated as SE. But SE can also mean many things: Sales Engineer, Software Engineer, Security Engineer, Support Engineer, Service Engineer, etc. And I learned from an IBMer many years ago, who was tasked to shift office (and thus earned the title Shifting Engineer), SE might as well mean Super Engineer. SEs do everything!

And honestly, what we really need here are Super Engineers too. Our SEs crawl around to lay cables, carry heavy equipment, make UTP patch cords, get their hands dirty assembling and disassembling equipment, climb rooftops and investigate plumbing problems, climb over mountains (okay it’s just a hill…) to follow cable trenching works, etc. Yes, there are certainly some work hazards too. But remember the benefits? That’s where you get to learn! Where do you get to be involved in such varied work?

Our operating culture here is that we often build rather than buy. Hence, many systems and applications are built ourselves. We build on many open source technologies (e.g. LAMP) and program in Perl. We run lots of Cisco hardware, but we don’t really consider ourselves a Cisco shop… because we do continue to procure other brands of products. We run two data centres with state-of-the-art facilities monitoring and water cooling. And while traditionally many network teams cannot get along with security teams… well, we’re the same team, which means we get to run security efficiently and effectively too.

Alright, if all these sounds interesting to you, let me know. I’ll drop you a formal recruitment advert.

Related posts:

1. Hiring More Super Engineers We’re still looking to hire more staff to support...
2. Recruiting Student Network Engineers We’re starting to recruit student helpers again. They will help...
3. Fire Suppression Gas Discharged We witnessed the discharge of HFC227ea (FM200) fire suppression...

Well, my fingers were itchy, and I wanted to test something out. Something that carried a small but inherent risk of mucking up. I was careful, no less, to make sure there would be no mistakes, no typos. In fact, I wrote a “rollback” script to undo the changes after a timed delay, just as a precaution. So, even if the changes mucked up the system, the changes would be automatically rolled back anyway. Then, how about testing this rollback script too? Ah, that’s where it screwed up. Something else was broken, and the rollback script got stuck, and it disconnected my server from the net.

And so in the dead of the night, I had to make a trip down to the colo data centre to fix my server. The security guard was delighted to have some company. I wasn’t. Well at least not in the wee hours of a Monday morning. He was in the mood to chat, and I tried my best to be polite.

The best way to keep a server running “forever” is to not touch it at all. Itchy fingers crash servers. Reboot a server, and maybe it never comes back up. (Oh yes, I have seen that many times.) So far in my experience, about the only things that spontaneously break down on their own are hard disks.

Just for the interest of those who have not seen real servers, the above picture shows where this website actually lives. It is a rackmount server measuring about 17″ wide, 1.75″ high. This is what the industry refers to as a “1U” rackmount server. The “U” is a measurement of “rack unit”, 1,75″ high, corresponding to standard dimensions of equipment that these equipment racks are designed to accommodate. Equipment depths can vary widely, but rarely exceed 30″ deep. Although I run data centres too, I pay to rent space in someone else’s commercial data centre to host this website.

I was looking around this colo data centre and observed how many other customers still don’t install proper rackmount servers. The most common “servers” are actually PCs contained in desktop or tower chassis (casing). In fact, many “well known” Singapore websites are hosted in this place using clunky tower PCs. You could maybe house 4 or 6 of these tower PC boxes in a standard equipment rack. If you had a proper 1U rackmount server (i.e. like the one pictured above), you could fit 42 of them in a standard equipment rack. It would save you tremendous amount of space.

Data centre space is extremely costly. It costs anywhere from S$5K to S$10K per square metre to turn an office space into a data centre. Yes, that excludes the cost of building up the room (i.e. the building structure such as the floor, walls and ceiling). As a result, rental of rack space can be quite costly. This was the case in the dot-com boom.

Rental costs have dropped since the dot-com bubble burst, of course. But the real cost of building a data centre remains high.

Anyway, here’s how a blade server looks like. As data centre space is precious, there is this continual drive to shrink servers. Pictured here are 2 blade chassis, accommodating a total of 28 servers. Total height is only 24.5″ (or about 63cm). Mind you, these are very high-end powerful servers. Not some miniaturized classic Pentium PC. How’s that for space efficiency?

Related posts:

1. Hot Data Centre, Very Very Hot Data centres pack lots of cool technology in them. The...
2. Moving To New Data Centre Last weekend, zitseng.com and the rest of my servers were...
3. Water-Cooling the Modern Data Centre We have a data centre that recently became the first...

We have a data centre that recently became the first in South East Asia (or perhaps in Asia too) to operate a certain type of liquid cooled rack. Liquid cooling technology is actually not new; It was quite common maybe 2 decades ago. The large mainframe computers of that era produced so much heat that they needed to be cooled by chilled water or refrigerant. Liquid cooling has become less common since then. Nowadays, the thought of water in a data centre is unimaginable to many data centre operators. But today, the advent of very high density computing is beginning to re-introduce liquid cooling into data centres.

After the mainframe era, the trend was for computers to become smaller, consume less power, and produce less heat. Today, computers have indeed become smaller very rapidly, but the trend for power consumption and heat dissipation doesn’t seem to have followed. Powerful computers have shrunk into 1U form factor (just 1.75″ height), and then further into “blades” that are slotted into chassis. We have gone from single-core CPUs to quad-core CPUs. Today, we have actual production server racks with 15KW of load, and mind you that’s actual load (as opposed to rated specifications). That is equivalent to like 15 ovens in a 1000mm x 600mm floor space!

Heating has been a perennial problem in our data centres. At one of our old sites, our greatly oversized (so we were told) air conditioning systems just could not maintain an ambient room temperature of 28 degrees celcius, let alone the “ideal” 22 degrees celcius we would love to achieve. When we finally moved out of that data centre, we observed that once all the servers were gone, the room temperature dropped to as low as 10 degrees celcius! We did indeed have damned powerful cooling! But it was no match for modern day high density computing.

One of the most unusual sights in our latest data centre is the network of huge chilled water pipes running under the raised floor system. While others are keeping water out of their data centre, we have ours filled with water pipes. It is not so bad if you use refrigerant, because if the pipe springs a leak, the refrigerant simply evaporates and you are left with little mess. With water, of course, a leak could mean a flooded data centre. Water don’t mix well with electricity, the reason why most sites are very eager to keep water out of the IT server area.

Why chilled water? It turns out that chilled water is a lot more effective and efficient at carrying away heat than refrigerant. Most water-cooled solutions at present, however, are focused on bringing chilled water to a cooling coil nearer to the heat source, rather than using the chilled water to directly cool the heat source. The problem with the latter is that significant engineering changes are required for severs, storage and other IT equipment to accommodate water circuits.

Now if you are thinking about those water heat sinks and radiators you have come across in Sim Lim Square… this is not quite the same thing. Those Sim Lim Square PCs do not need water. The fanciful cooling is just that: fanciful. They typically don’t even need air-conditioning.

Related posts:

1. Hot Data Centre, Very Very Hot Data centres pack lots of cool technology in them. The...
2. Wow! Is that a Mainframe Computer? [photopress:20071005535.jpg,thumb,pp_image]A computer science student saw this, and asked if it...
3. Moving To New Data Centre Last weekend, zitseng.com and the rest of my servers were...

[photopress:20071005535.jpg,thumb,pp_image]A computer science student saw this, and asked if it was a mainframe computer. I was almost going to burst out laughing, but managed to hold myself together. I’m sure most of you know it is not a computer at all. This 1.7m high contraption is an isolation transformer. At least you should be able to tell it is some kind of a transformer, right?

I just thought to myself that, my goodness, our students don’t even know how a mainframe computer looks like? Not even seen anything from photos or pictures? Mainframe computers are no doubt getting quite rare these days, so it is not surprising that few people have actually seen the real thing with their own eyes. But I’d imagine at least a computer science student would have some impression from pictures in books or on the Internet. (Check Google images for some impressions.) The last mainframe at my workplace, an IBM 3090, was decommissioned and removed before year 2000 (I think).

It’s just amazing how technology has progressed. When I started work, VAXes were still in operation. The trend at that time was for things to become smaller, more compact, consume less electricity and generate less heat. Then, when high performance cluster computing became more prevalent, some of these trends started to reverse. Our Data Centres started to demand more electricity, and cooling became a major challenge again. More than 10 years ago, we needed 32A 3-phase power supplies, then it became alright to cut down to 32A single-phase power (and indeed many commercial data centres simply standardize on 16A single-phase power), but today we are back to 32A 3-phase and even 63A single-phase power supplies. Data Centres used to have lots of plumbing (to carry chilled water or refrigerant for cooling mainframe computers), then CRACs (computer room air-conditioning, which are just expensive air-cons that can control temperature and humidity) became standard, and guess what? Today people are talking about water-cooled CPUs again. Indeed, my latest Data Centre now has an underfloor plumbing system carrying chilled water into liquid cooling racks.

Another student passing by gave an estimate that this contraption (the isolation transformer) weighed maybe 1000kg. Yes it is heavy, but no, nowhere close. More like 4 tons perhaps. Ok, I admit, estimating weight is not so easy. Although most people know that metal is heavy, they may not appreciate how much heavier a humongous chunk of metal like this would weigh. (Interestingly, I was surprised at the reality of how heavy water is. You know 1 cubic meter of water weights 1 ton? This is from physics lesson in school, yah? Just imagine, 1 cubic meter is just 1m x 1m x 1m only, and that little cube is 1 ton!)

Well, anyway, we were replacing a 1.2MVA isolation transformer. It took our movers all of 3 hours to remove the transformer above. Just move on same level, no steps, under 100 meters, rolling all the way. Imagine the effort to push 4 tons. Okay, some of the delay was due to floor protection (the need to lay plywood and steel plates to both protect the floor finishing and distribute the load).

Related posts:

1. Water-Cooling the Modern Data Centre We have a data centre that recently became the first...
2. Hot Data Centre, Very Very Hot Data centres pack lots of cool technology in them. The...
3. Pickup A Free Notebook Computer If you’re looking for a fantastic free notebook deal, I...