Zit Seng's Superwall » servers

Second Life for Our Blade Servers

Zit Seng — Fri, 23 Sep 2011 01:36:15 +0000

No, not the game Second Life, but trying to give a second lease of life to old abandoned hardware. Recently I noticed a big bunch of IBM blade servers lying round, dismantled, and stacked on the floor. I thought it would be good to take over them and use them for some student projects. So I went around to seek blessings to officially hijack the blades. I didn’t really know what was in the blade, or their hardware configuration, and I was told that they might not even be working.

Nevertheless, with fingers crossed, we proceeded to help ourselves to one chassis and 14 blades. It’s the H-series IBM Blade Center chassis, with HS20-series blade servers. Yeah, they sound old. But we’re not spending any money. If you don’t fork out money, you don’t get to choose.

The first thing we did when we laid hands on the blade servers was to open it up to inspect what was in it. Two CPU sockets, seen clearly in the foreground of the photo above. We can’t tell what CPUs they were, since they were covered by the heat sink. The RAM modules, 4 sticks of them, turned out to be a miserly 256MB per piece. So that’s just a total of 1GB. 1GB of RAM is like the amount we allocate to a virtual machine instance nowadays. Then the disk? 40GB. IDE. Really low-end old hardware.

After booting up the blade servers and putting Linux in it, we saw what the CPU was. Xeon. 32-bits. They had 2 processor cores per socket, and with hyperthreading, so there were effectively 8 processing cores in total. The CPU ran at 2.8GHz.

The hardware specifications weren’t all that great. Most people nowadays will have desktop computers that are more powerful than this. My desktop computer, which I upgraded recently, has a 64-bit 3.1GHz CPU with 4 cores and 16GB of RAM. You can’t tell much from GHz nowadays, but I think mine would spin circles around this IBM blade server.

But I imagine the whole experience of playing with the servers, setting them up, etc, would be fun for our students. How many people get the opportunity to work with blade servers? They aren’t cheap. They aren’t cheaper than the typical 1U rackmount servers, so for simple server requirements, SMEs are probably not going to be willing to fork out money for blade servers.

Most importantly, these blade servers are living a second life. They didn’t cost us money. (Okay, I know we did have to pay for them many years ago… but if you look at accounting principles, the assets have been fully depreciated by now, so it is “free”.)

How I Crashed My Website Well, my fingers were itchy, and I wanted to test...
Work-Life Balance Sometimes, I think “work-life balance” is just an excuse for...
A New Lease of Life for the Nokia N97 I couldn’t wait any longer, so I did what I...

Picking Up PostgreSQL

Zit Seng — Sat, 28 May 2011 06:19:54 +0000

I’ve been a MySQL user for a long time. But since many years ago, I’ve been thinking about switching to adopt PostgreSQL, partly because how it looked like MySQL was headed toward commercialization (and indeed as we all know now they are owned by Oracle). What attracted me to PostgreSQL was the relative purity and completeness of its SQL implementation, and lesser sense of commercialization in its project.

PostgreSQL has generally been reputed to have had a lot more focus on robustness and data integrity. This is important. In the early ages of MySQL, there was no ACID support, which was a major bummer to me.

However, in those early ages, PostgreSQL was slow, and MySQL was blazingly fast. If you’re developing web application, speed was important, and the lack of ACID support could be compensated through application algorithms. MySQL caught up with some of PostgreSQL strengths, but PostgreSQL was slow in catching up with those of MySQL.

A month or so ago, I re-examined PostgreSQL again. I was looking at implementing Request Tracker 4 (RT4) with fulltext indexing. It was supposed to support MySQL, but apparently because of a bug in RT4′s code, I couldn’t get it setup. So, as I observed in the mailing list about people moving to PostgreSQL (for reasons possibly not related to this), I decided to try out PostgreSQL again.

Not too long before, I was also evaluating MySQL Cluster. For me, this is the next big thing to do. We can build highly robust high-availability infrastructure for network, load balancers, servers, and web applications, but it all comes to nought when you realize the database was a single-point-of-failure. MySQL Cluster would be the answer to this problem. MySQL Cluster isn’t all that new. But in its earlier incarnations, it had a couple of difficult constraints that all its data needed to fit in RAM. If you had a 16GB-sized database, you needed a server to contain all 16GB of it, or 2 servers of 8GB if data was partitioned into two groups. Although nowadays servers with 16GB of RAM are commonplace, it was still a capacity limitation that I didn’t like. MySQL Cluster evolved, and now only its indices need to fit in RAM.

Then, RT4 came along. My objective to setup fulltext search incides meant I had little choice but to turn to PostgreSQL. It’s quite a bit of a learning curve. The way authentication and authorization is done is different from MySQL. But it’s not something that will take you more than a few hours to figure out and have everything setup nicely.

PostgreSQL, unfortunately, still doesn’t have decent out-of-the-box solutions for high-availability clustering. Version 9.1, currently in beta, will have support for synchronous replication, but it still isn’t quite the same thing as the high-availability solution offered by MySQL Cluster.

Scaling Out CodeCrunch I recently found myself leading an application system project: CodeCrunch....
Wow! Is that a Mainframe Computer? A computer science student saw this, and asked if it...

Brand New Yet Used Hard Disk

Zit Seng — Fri, 20 Nov 2009 09:28:23 +0000

We bought a brand new branded computer. It came from a reputed brand vendor (or also called a system integrator). We would expect it to come with brand new components. One of them was a branded hard disk. The hard disk died. That’s nothing too unusual. Hard disks are one of the most likely components to fail because it is a mechanical part. However, we were in for a surprise when we removed the failed hard disk to do a replacement.

Our branded hard disk, in a branded rackmount server, bought from a brand name company, was in fact a Certified Repaired HDD. In other words, it was a used, or refurbished hard disk! Wow!

Now we see that it is no wonder that the hard disk failed. Well, of course, even a refurbished hard disk would have been subject to properly quality checks and in should work pretty well too. I guess we just wanted to blame something for the problem. Furthermore, we’re just so surprised about getting a refurbished hard disk when we bought a brand new computer. I don’t think anyone would be very pleased about this at all.

You can see from the photo. This is a Seagate hard disk from an IBM eServer xSeries. It’s “date of manufacture” is a pretty long time ago.

New Hitachi 1TB Hard Disk My not terribly old 320GB Seagate hard disk thought it...
Fixing a Dying Hard Disk My (wife’s) desktop PC at home started to act up....
Pre-Comex Shopping Another item I acquired during my pre-Comex shopping trip last...

Death Of A Server

Zit Seng — Wed, 03 Sep 2008 02:44:57 +0000

One of my servers died recently. A few days prior to that, there was something very odd with its clock, which was stuck between 03:14:31 and 03:14:35. Then on its last day, it suffered a kernel panic. After a hardware power cycle, the server never came back up. It’s like it suffered a heart attack, and CPR and defribillation failed to resuscitate it.

Well, this was an old hardware, probably almost 8 years old. That’s somewhat aged considering that nowadays desktop computers are changed every 3 years and servers maybe lasting just slightly longer. You might have heard of Moore’s Law. This is not what the original Moore’s Law states, but basically performance will double every 18 months, or the price will be halved. It’s no wonder that it makes sense to upgrade our tech toys frequently.

In fact, nowadays the ideas pushed by the green people are also saying that old computers consume too much power. With rising oil prices, the cost of keeping an old computer running might exceed the cost of buying a new one, particularly also considering the increased performance of newer hardware (and hence possibly reducing the number of new hardware required).

Once upon a time, servers had personalities. At least, (crazy) system administrators make pretend that their servers have a mind and soul. But nowadays, they are just numbers in a spreadsheet: performance benchmark scores, total cost of ownership, cost-per-SPECmark, cost-per-GB, etc.

Second Life for Our Blade Servers No, not the game Second Life, but trying to give...
GreenGeeks vs Bluehost Review This is my in-depth technical review and comparison of GreenGeeks...
Power Failure and a Bad Weekend This Saturday was not a good day. I went back...

How I Crashed My Website

Zit Seng — Mon, 29 Oct 2007 05:29:07 +0000

Well, my fingers were itchy, and I wanted to test something out. Something that carried a small but inherent risk of mucking up. I was careful, no less, to make sure there would be no mistakes, no typos. In fact, I wrote a “rollback” script to undo the changes after a timed delay, just as a precaution. So, even if the changes mucked up the system, the changes would be automatically rolled back anyway. Then, how about testing this rollback script too? Ah, that’s where it screwed up. Something else was broken, and the rollback script got stuck, and it disconnected my server from the net.

And so in the dead of the night, I had to make a trip down to the colo data centre to fix my server. The security guard was delighted to have some company. I wasn’t. Well at least not in the wee hours of a Monday morning. He was in the mood to chat, and I tried my best to be polite.

The best way to keep a server running “forever” is to not touch it at all. Itchy fingers crash servers. Reboot a server, and maybe it never comes back up. (Oh yes, I have seen that many times.) So far in my experience, about the only things that spontaneously break down on their own are hard disks.

Just for the interest of those who have not seen real servers, the above picture shows where this website actually lives. It is a rackmount server measuring about 17″ wide, 1.75″ high. This is what the industry refers to as a “1U” rackmount server. The “U” is a measurement of “rack unit”, 1,75″ high, corresponding to standard dimensions of equipment that these equipment racks are designed to accommodate. Equipment depths can vary widely, but rarely exceed 30″ deep. Although I run data centres too, I pay to rent space in someone else’s commercial data centre to host this website.

I was looking around this colo data centre and observed how many other customers still don’t install proper rackmount servers. The most common “servers” are actually PCs contained in desktop or tower chassis (casing). In fact, many “well known” Singapore websites are hosted in this place using clunky tower PCs. You could maybe house 4 or 6 of these tower PC boxes in a standard equipment rack. If you had a proper 1U rackmount server (i.e. like the one pictured above), you could fit 42 of them in a standard equipment rack. It would save you tremendous amount of space.

Data centre space is extremely costly. It costs anywhere from S$5K to S$10K per square metre to turn an office space into a data centre. Yes, that excludes the cost of building up the room (i.e. the building structure such as the floor, walls and ceiling). As a result, rental of rack space can be quite costly. This was the case in the dot-com boom.

Rental costs have dropped since the dot-com bubble burst, of course. But the real cost of building a data centre remains high.

Anyway, here’s how a blade server looks like. As data centre space is precious, there is this continual drive to shrink servers. Pictured here are 2 blade chassis, accommodating a total of 28 servers. Total height is only 24.5″ (or about 63cm). Mind you, these are very high-end powerful servers. Not some miniaturized classic Pentium PC. How’s that for space efficiency?

A Very Very Hot Data Centre Data centres pack lots of cool technology in them. The...
Water-Cooling the Modern Data Centre We have a data centre that recently became the first...
Moving To New Data Centre Last weekend, zitseng.com and the rest of my servers were...

Wow! Is that a Mainframe Computer?

Zit Seng — Sat, 06 Oct 2007 01:31:41 +0000

A computer science student saw this, and asked if it was a mainframe computer. I was almost going to burst out laughing, but managed to hold myself together. I’m sure most of you know it is not a computer at all. This 1.7m high contraption is an isolation transformer. At least you should be able to tell it is some kind of a transformer, right?

I just thought to myself that, my goodness, our students don’t even know how a mainframe computer looks like? Not even seen anything from photos or pictures? Mainframe computers are no doubt getting quite rare these days, so it is not surprising that few people have actually seen the real thing with their own eyes. But I’d imagine at least a computer science student would have some impression from pictures in books or on the Internet. (Check Google images for some impressions.) The last mainframe at my workplace, an IBM 3090, was decommissioned and removed before year 2000 (I think).

It’s just amazing how technology has progressed. When I started work, VAXes were still in operation. The trend at that time was for things to become smaller, more compact, consume less electricity and generate less heat. Then, when high performance cluster computing became more prevalent, some of these trends started to reverse. Our Data Centres started to demand more electricity, and cooling became a major challenge again. More than 10 years ago, we needed 32A 3-phase power supplies, then it became alright to cut down to 32A single-phase power (and indeed many commercial data centres simply standardize on 16A single-phase power), but today we are back to 32A 3-phase and even 63A single-phase power supplies. Data Centres used to have lots of plumbing (to carry chilled water or refrigerant for cooling mainframe computers), then CRACs (computer room air-conditioning, which are just expensive air-cons that can control temperature and humidity) became standard, and guess what? Today people are talking about water-cooled CPUs again. Indeed, my latest Data Centre now has an underfloor plumbing system carrying chilled water into liquid cooling racks.

Another student passing by gave an estimate that this contraption (the isolation transformer) weighed maybe 1000kg. Yes it is heavy, but no, nowhere close. More like 4 tons perhaps. Ok, I admit, estimating weight is not so easy. Although most people know that metal is heavy, they may not appreciate how much heavier a humongous chunk of metal like this would weigh. (Interestingly, I was surprised at the reality of how heavy water is. You know 1 cubic meter of water weights 1 ton? This is from physics lesson in school, yah? Just imagine, 1 cubic meter is just 1m x 1m x 1m only, and that little cube is 1 ton!)

Well, anyway, we were replacing a 1.2MVA isolation transformer. It took our movers all of 3 hours to remove the transformer above. Just move on same level, no steps, under 100 meters, rolling all the way. Imagine the effort to push 4 tons. Okay, some of the delay was due to floor protection (the need to lay plywood and steel plates to both protect the floor finishing and distribute the load).

Water-Cooling the Modern Data Centre We have a data centre that recently became the first...
Seizing Control of Vanessa’s Computer One of the ways to keep Vanessa occupied is to...
My Best Computer Mouse Ever The batteries on my Apple Wireless Mighty Mouse finally died,...