I recently found myself babysitting a certain web application project as it went live. It was besieged with many performance problems as it scaled up to meet real-life usage. This is a pretty common problem. I deliver a guest lecture from time to time on the topic of building scaleable web application infrastructure, and although this is something pretty familiar to me, I can see that many people are bewildered by the complexities of real-world production web application infrastructure.

The challenge of building scaleable web application infrastructure is that you need domain knowledge that cuts through all layers of the OSI model (or the TCP/IP model if you prefer). Most of the time, application developers are the main drivers of web application projects, but they don’t know much about operating systems, server hardware, storage, and networking. It is usually not their job to know all those things, because there are other people to take care of them.

However, there is a big divide between the people who operate above the operating system level, and the people who work with the operating system or below. The divide often means the two sides don’t communicate on the same terms with each other, and they don’t appreciate the problems experienced by the other.

So, anyway, what’s interesting this time around is about syscalls. A syscall (or system call) is a call to an operating system function. It is below the user library, or the common system libraries that still run in “userland”. A syscall is more expensive than an ordinary function call because there is the matter of “privileged transfer” into operating system code.

I was looking into this Drupal application to find out why it was sucking up so much CPU. Web applications are usually not really CPU-bound. But this problem was rather unusual. So I started to strace the httpd processes. I found this drupal application made over 46K syscalls to service a single web request. Sounds horrendous.

I know 46K is too many syscalls, but I forgot what is a “right number”. I have a site running on Drupal, and I tested that. This site made over 6K syscalls. That’s almost an order of magnitude less. But even then, I thought 6K is still plenty.

For many years, we have been developing our own network management portal. It has grown into a mammoth application, and it has also gotten very old and dated. I tested its mod_perl-based web-front. Nice… just over 500 syscalls to service a single web request. As I looked through the 500 syscalls, I realize that many “wrong things” have crept into our code, but nevermind, 500 syscalls is already an order of magnitude lesser than our better Drupal site.

I next tested a WordPress website I have. It’s actually the old instance of this blog. Nice, 700+ syscalls per request. I’m suddenly not liking Drupal, and liking WordPress a lot more. WordPress does deserve its CMS-of-the-year award.

Just to put things in perspective, the number of syscalls isn’t anywhere the most important metric to look at. Not all syscalls are equal either. Syscalls that result in disk I/O are generally going to be more expensive (time-wise) than one that, say, fetches the system time. This is something worth looking at when you’re trying to squeeze the last drop of performance out of the app.

Related posts:

1. The New Operating System Virtualization is not new to me. I’m now using a...
2. N95 Apps to Annoy Your Friends Here are some applications for your Nokia N95 to annoy...
3. Building My New PC It’s been quite a long while since I last built...

We had pushed out CodeCrunch with the intention that it will one day replace an older automated programming assessment system we’re running. The two most important advantages of CodeCrunch are: web-based user interface, and system designed to scale out.

Everyone’s familiar with using web browsers, and so clearly any web-based system is going to be much preferred over another that requires client-side software installation. No need to elaborate anymore here.

The other advantage about scalability is rather fascinating. One of the limitation of the other system (and possibly many other simliar systems used elsewhere) is that it cannot easily scale to accommodate more demanding workloads. Think about a programming lab exam where there might be 500 students submitting programs for automatic assessment. Sure, you deploy more power hardware, faster CPUs, more RAM, etc. But in our past experience, the achilles heel is working with Java programs. How powerful a system do you need to support, possibly, 500 concurrent Java compilations and testing sequences? What if each JVM required a few hundred MBs of memory?

The only solution, really, is for the system to scale out to a distributed architecture. Better yet if the workload can be scheduled asynchronously. This is, in my opinion, the biggest strong point of CodeCrunch’s system architecture.

Today, the CodeCrunch cluster comprises:

• One cluster head node: Database, scheduler, and central storage.
• 6x worker nodes: These are the compute nodes doing the grunt program compilation, testing and assessment.
• 2x web server nodes: The web servers have very light load, but we run two simply for redundancy purposes.

The cluster is fronted by a pair of load balancers, which also serve to terminate and accelerate SSL connections. On top of the failover/high-availability provided by the load balancer, the SSL termination is particularly helpful, because it relieves the web servers from encryption/decryption overheads. (Our load balancer also performs on-the-fly content compression.)

The distributed design allows the system to scale out to additional nodes rather trivially if required by the workload demand. The nature of asynchronous job scheduling also means that the system will not work itself to death if, for any reason, it cannot keep up with the workload thrown at it.

An interesting phenomenon of web application systems is that, if not properly designed, they often run into a danger of killing themselves as they near peak capacity. Here’s how it typically happens. As the rate of web requests increase, the response gets slower because of CPU bottlenecks, storage I/O bottlenecks, database lock contentions, etc. As response gets slower and slower, at some point, users get “fed up” and click reload on their web browser. The web server doesn’t necessarily know that the previous request has been “abandoned” and simply sees the reloads as additional web requests, thus further worsening the situation. If not properly managed, the problem will simply escalate exponentially until the site kils itself.

The problem is actually so easy to solve as to simply add more nodes to a distributed system. At some point, your distributed system will also get overloaded. The challenge is really to get the system to “fail” gracefully.

CodeCrunch’s nodes are, actually, not real physical hosts. They are virtual machines running on blade servers (like the one pictured above). This allows us to easily move VM instances from a more busy physical host to a less busy physical host when needed. It also allows us to easily manage hardware failures. Better yet, we can even easily spawn additional VM instances during peak usage, or shutdown some VM instances and consolidate others into fewer physical hosts during lull periods. Oh yes, that’s what some vendors are selling as being “green”.

We use MySQL for our database, and like most MySQL users, we have not gone the path of MySQL Cluster. This means that the database is both a potential performance bottleneck and a single-point-of-failure. Not too urgent at this time, but at some point we’ll need to look into this.

Another concern is with storage. There is currently a single central storage volume. We can easily the volume size without too much trouble. In fact, I just did that this morning (JADDOG’s guide was helpful). But this central storage volume is a single-point-of-failure.

Not all problems need to be fixed, of course. The MySQL and storage are concerns, but it may turn out that solving them may not be worthwhile for our purposes.

Related posts:

1. Building Scaleable Web Apps – Syscalls I recently found myself babysitting a certain web application project...
2. Moving on to Solid-State Drives I think the best upgrade you can do for your...
3. Cold Storage is Charged This Cold Storage is charged. With static electricity, that is....

Ever since I signed up for broadband data on my mobile phone, I’ve been trying to make it easy for me to access information on the go. One of the things I’ve done is to write a simple widget to fetch little bits of information from different sources and mash it up into a tight little display area. For example, I have displays to show information from work, webcam feed from home, etc. One page that will probably find general interest to other people as well is the display of traffic information, weather forecast and CNA front page news. I’ve since cleaned it up and moved this to a public server so that it can be shared with everyone.

If you’ve got a mobile broadband plan on your phone, point your phone’s web browser to http://zitseng.com/widget. The page is plain and simple, so that it loads fast and minimizes unnecessary clutter and distraction on your phone. It is rendered in XHTML Mobile profile to maximize compatibility with most phone browsers (web browsers required though, not WAP-only browsers), and it will also display on all modern PC (Windows, Mac, Linux, etc) browsers as well.

The data displayed is fetched from different sources on the web. Traffic data is from One Motoring, weather is the 12 hour forecast from NEA, and CNA news is from CNA of course. When you access the widget, the local server cache is checked for recent copies of data. If they are too old, new data is fetched from the origin site in real time. The timestamp of the data is displayed so you can see how fresh it is.

This is a web widget, which means there is nothing to install in your phone. It is just a webpage, and bookmarking it is all that is needed to “install”.

It’ll be cool if a web page can extract cell tower information like how the Google Maps mobile application does, in order to deliver location-aware information. I don’t know if this can be done through a web-based application?

Related posts:

1. Nokia Secret Codes This page lists a couple of useful and/or interesting “codes”...
2. Shaming Idiotic Drivers I wonder what has happened with Parking Idiots blog? Some...
3. Boost In Google PageRank I wonder if Google has become more generous with their...

The obvious solution for documentation to a netizen is a wiki. There are plenty of wiki software. I have specific requirements too. Yes, functional requirements apart from the fact that the software has to be, hopefully, beautiful. Basically, I want a system with  good functional WYSIWYG editor and supported by a MySQL backend storage. It needs to integrate images, file attachments. It should be somewhat mainstream, actively supported or have an active development community. All these quickly narrows down the candidate list to almost nothing. Okay, let’s leave out beauty and see what we’ve got. Still, nothing stands out.

Fine. I start thinking about other solutions. The idea of a CMS portal soon comes to mind. After poking around a selection of CMS platforms, I eventually started to focus on Joomla and Drupal. Yes how uninteresting, it’s what most people try to choose between too.

Joomla is beautiful. Drupal is ugly. But for what I want to do, it seems Drupal works better. It has better capabilities to handle a site that will be managed by a bunch of different people and supported by a community of contributors. I’m not trying to start a Joomla vs Drupal debate here… so let’s just leave it at this. Bottom-line is that I like how Joomla looks, but more how Drupal works.

The next item of consideration is the WYSIWYG editor. The two top choices that come into consideration are TinyMCE and FCKeditor. TinyMCE is ugly. FCKeditor looks cool.

But I discover an annoying problem with FCKeditor. It outputs HTML (that gets sent back to Drupal) that concatenates all the text into continuous lines. Sure, as far as HTML is concerned, whether you actually put line breaks or not, the rendered output is still the same (except of course within certain types of HTML tags that mark up pre-formatted output). TinyMCE doesn’t have this problem.

I need the WYSIWYG editor to produce formatted HTML output that has line breaks at suitable places. This is to make sure that lines don’t run too long, and they are broken up at predictable places, and so that diffs between versions are easier to to digest.

So again, I have to choose TinyMCE. But I guess this is a bug that FCKeditor folks can fix, and I certainly hope they do.

Related posts:

1. Learning To Blog One of the cute little things that Vanessa tries to...
2. Copycat Logo I was coming home a couple of days ago when...
3. Teaching People to Work Together It’s funny how when we are small kids, we don’t...

Related posts:

1. Urgent WordPress Security Update Security updates keeps us busy during festive holidays again. The...
2. WordPress 2.5.1 Security Update WordPress 2.5.1 was just released yesterday to fix one very...
3. WordPress 2.6.5 Security Update WordPress has been updated to version 2.6.5, which fixes one...