IPv6 network connectivity on my home fibre broadband service is broken at the moment. I’m definitely not happy about it. So perhaps I’m mostly just ranting when I complain about their network engineering being a failure. But it really has to be partly true. The fact that the problem is not being proactively detected and rectified behind the scenes certainly reflects very badly on their network management capabilities.
That the hopelessly lost customer service folks whom I had to deal with just makes things worse. Sometimes I’d wonder if the purpose of helpdesk is to frustrate customers enough so that they give up. The problem then goes away. Ticket is thus considered resolved. No, the problem didn’t actually get resolved; it just went away.
I’ve been quite excited about IPv6. I used to have Super Internet, and I was really pleased with their IPv6 service which they rolled out pretty early. I would have loved to write nice things about StarHub’s IPv6 service. Unfortunately, StarHub’s IPv6 breaks regularly. I used to not mind too much about it. But nowadays I notice it a lot more, and mind much more when it breaks.
Before I launch into my tirade about how lost StarHub is with the service they are providing, I want to say a few things about the stupidity of their customer service setup. When you call 1633, somewhere before you get to speak to the human, you are required to key in your NRIC number (or FIN, or some other identification number). Once you get a human on the line, one of the very first thing the person will ask for is your NRIC number. Erm, isn’t it already captured by the system? Damn. Why do you ask for a piece of information that has already been asked for? If the system can’t make use of the information, then don’t ask for it! When you’re in a good mood, this is perhaps just a curious bit of inefficiency. When you’re not happy, this is just plain annoying.
I called StarHub customer service multiple times today. I learnt an interesting thing. For some stupid reason, despite having given my NRIC number twice, like it is such an important thing, my call was apparently “indexed” by the calling phone number. That’s how it seems to me, because in my second call today, the customer service fellow could not find information about my earlier call. I had to point out what time I called, and that I called from a different number, before he found information about it. Why, oh why, does it make sense to index by phone number when you already have the customer’s identity?
Let’s talk about my IPv6 problems now. A few moons ago, when I’ve had enough problems to want to bug StarHub about it, I considered various support options. I could call their hotline, 1633, but I thought those customer service folks would be quite lost. In the end, I decided to try their StarHub Community forum, in which it seemed some knowledgeable StarHub technical support people lurked around.
Big mistake. The first response I got, from a support staff whom I thought appeared quite knowledgeable, was that he didn’t think IPv6 was available. Oh wow. IPv6 was proudly advertised on StarHub web pages. Their staff did not know about it. I think StarHub should consider properly training their staff about their products and services.
Then, that fellow, whom I realised was not as knowledgeable as I had expected after all, asked about my need for IPv6 addresses. Oh my goodness. Is it any of his business? Perhaps he should have just asked, why do I want to connect to the Internet? Damn.
I’m asking about a service that is being marketed by StarHub. It is broken. The priority should be to troubleshoot, and fix. Why does it matter what I want to do with IPv6?
So well, IPv6 has been pretty wonky from time to time. Today, or last night, it broke again. This morning, I decided I needed to complain. I called 1633. Not very helpful, put on hold for a long while, before announcing that they will have someone call me back. No one called.
By the third time I called, I was getting pretty angry. I made it clear to the third customer service fellow that I was not happy having to bug them three times without the courtesy of them just returning my call with a useful update.
Finally, someone called. They wanted to dispatch someone to my house to troubleshoot the problem, presumably with my network. Erm. Excuse me. I made it pretty clear the problem was with their network. Not mine.
Let me explain a little bit. The Internet is a really bewildering web of networks. Networks connect to networks many times over. How does traffic from your computer reach the right server on the Internet? That happens with routing tables in the network. For your network to work, the Internet needs to know about it. In order words, your network must appear in the Internet routing table.
What I found happening today is that the IPv6 prefix delegated by StarHub’s DHCP server to my home network is not reachable on the Internet. This has nothing to do with my home network, my broadband router, my PC, or anything about me. A part of StarHub’s own network was missing from the Internet routing table. This is easily verifiable by checking public BGP4 Looking Glass tools.
Let me repeat this. A part of StarHub’s network is missing from the Internet routing tables. No one in the world can reach this part of StarHub’s network because there is no information about it in the routing tables.
Is this so difficult to understand? This is something, I told the StarHub technical staff who finally returned my call, that needs to be resolved by the core network engineering people. It needs to be escalated to those people so that they can sort out the problem. Oh dear, he told me that will be the “backend team”, and that they had said to send someone down to my home to troubleshoot.
Well, if that’s the case, I’m sure I’m quite qualified to say StarHub’s network engineering is a horrible fail.
I want to give them the benefit of doubt. Surely that “backend team” isn’t the true core network engineering folks. It will be really sad if the core network folks cannot understand routing. Oh wait. Actually, it is already sad that such an important thing is not being proactively monitored. I suppose most of the time, people don’t know about IPv6, and no one notices about IPv6 problems to complain about it, so they get by fine.
I am contemplating making a fourth call for today. I hope to get lucky, but I don’t reasonably expect to see much progress.
For the Techies…
Some additional information for interested techies. StarHub hands out IPv6 prefix delegations within the 2406:3003:201b::/48 network. By using a BGP looking glass service such as this one from Cogent, you can see that there is (currently) no route information for this network.
My router also receives IPv6 address within the 2406:3003:101b::/48 network. This one works fine. You can compare the result produced by the BGP looking glass service.
While my router itself has working IPv6, LAN clients behind my router are being given IPv6 addresses that are unreachable on the global Internet.
Update (9 Apr 2014): This post seems to have garnered a lot of interest, with many people contacting me through various means. In the interest of keeping everyone updated, my IPv6 access problem has been resolved as of yesterday afternoon. Nevertheless, one can see on sites like Hurricane Electric’s Looking Glass page for AS55430 (StarHub) that there was indeed a significant dip in IPv6 announced prefixes yesterday. I’ll be pleased to post another time more details about troubleshooting these connectivity problems.