I like data. So when I said in my last post about QUIC and HTTP/3 that I didn’t have statistical data for comparison, I realised that I really need to get that sorted out. Gut feel is one thing, but if one could have objective data, that would be a whole lot better, isn’t it?
The challenge with testing through a web browser is that it is tedious process of clicking and recording the results. The timings given in the Web Inspector are accurate, but you would need to take into account caching effects. I would prefer to use CLI tools like wget or curl, but neither supported HTTP/3 as shipped in Ubuntu 22.04.
Well, it turns out one can compile curl with HTTP/3 support. There are several libraries it can link against. You can refer to this build guide for curl with HTTP/3. I originally built the quiche version but found, to my surprise, that curl behaved consistently worse with HTTP/3 than with HTTP/2 for the same website. Your mileage may vary, but I did find some anecdotal comments on the Internet about similar experiences with quiche, so I subsequently built curl with ngtcp2 and nghttp3. Unless otherwise stated. the curl I refer to in this post is the version built with ngtcp2 and nghttp3.
I cannot fully explain the results, but let me just first report the benchmark findings with curl. My test involves fetching the URL https://zitseng.com/ (that’s just that one request for an HTML page) from a test Ubuntu 22.04 client. The test is repeated 30 times, i.e. sample size is 30. The value that is measured is the wall clock time as returned by the time command used to measure how long curl took to run.
| HTTP/3 | HTTP/2 | |
| Mean | 222.43 sec | 223.867 sec |
| Variance | 108.19 | 299.22 |
The two-tail P(T<=t) value is 0.6963, so at 5% significance, we do not reject the null hypothesis.
In layman language, there is no significant difference between HTTP/3 and HTTP/2. Strange yeah? But it is interesting to note the wild variance seen in HTTP/2 test.
My next test fetches all the resources that is required to fully render my blog homepage https://zitseng.com/, i.e. including all CSS, JS, fonts, etc. There are a total of 38 resources that are needed, excluding resources that are not hosted on my own web server. Curl is provided the URL list, and given the -Z parameters which tells it to parallelise its requests. (Note that in real life, a web browser has to download the main HTML content first, then progressively figure out what other resources it needs to retrieve.)
| HTTP/3 | HTTP/2 | |
| Mean | 227.67 | 286.57 |
| Variance | 120.44 | 757.77 |
The two-tail P(T<=t) value is 3.0603E-13, i.e. it is so small, we very confidently reject the null hypothesis.
That, in layman terms, means that HTTP/3 is clearly faster. It’s quite evident from a cursory glance at the data.
Again, the variance for HTTP/2 request times is very high. I suppose the TCP session setup messed things up quite a bit. I did not check how many TCP sessions curl setup for a single test run.
An interesting observation here is that using HTTP/3, it appears to make little difference fetching a single HTML resource versus everything required to render my blog homepage.
Just out of curiosity, I performed the statistical test for the two HTTP/3 runs (i.e. fetching just the HTML of the homepage versus fetching all 38 resources required to render the homepage). The two-tail P(T<=t) value is 0.0585, which statistically says we are unable to reject the null hypothesis, and that in layman terms says that the two means are the same. (Aside: Trying to keep things simple here.)
Funny that fetching a whole lot of stuff works out to be roughly the same as fetching just one. Does this say something quite phenomenal about HTTP/3’s performance in parallel downloads?
I also did a quick test, though I don’t have the data now, to measure the performance of Apache vs Nginx with both running only HTTP/2. Nginx is quite clearly ahead of Apache. This is probably not surprising, since the huge lead Nginx+QUIC had over Apache that I mentioned in my last post is unlikely to be attributed to HTTP/3 (i.e. QUIC) alone.
Just for clarity, in both my Apache and Nginx setups, I use the exact same php-fpm to run the PHP scripts. In other words, Apache and Nginx are just the frontends that offload PHP to php-fpm.
I like to work on data, and I like to test with data. Many years ago, I did not find Nginx to be faster than Apache in hosting my WordPress blog. Today, things are quite different. Plain Nginx without HTTP/3 is already a win. Add in HTTP/3 support, it’s even better.
ps: Cover photo is of a data centre that is just something from my photo archive. My website is not hosted there.