I’ll be honest. There’s no way you’ll become a skilled network techie just by reading this blog post. After the recent loss of BGP route reachability to parts of StarHub’s network, I thought I’d share a little bit of network troubleshooting knowledge. This is no rocket science, but clearly it also isn’t something you’d grasp overnight, at least not unless you already have some basic network knowledge.
Let’s recap. On Monday, IPv6 access on my home fibre broadband broke. I know, most people aren’t going to notice these things. I do. The IPvFoo Chrome browser extension makes it extremely obvious to me whether the site I’m visiting is on IPv4 or IPv6. Facebook ought to be on IPv6, but it wasn’t that day. I verified by hand.
$ ping6 www.facebook.com
Yup, Facebook was not up on IPv6. But checking Facebook with the regular IPv4 ping was successful. So, something was broke. It could be that Facebook is broke, or it could be that my IPv6 is broke.
As a network techie, I don’t jump to the conclusion that someone else’s system or network is broke. The logical way of troubleshooting is to test from my end first. I know all these may be very basic for some people, but let me just go through a few troubleshooting steps.
- Does my computer have an IPv6 address? (Check with ifconfig.)
- Do I have a default route out of my network with IPv6? (Check with netstat.)
- Can I ping the gateway of my default route? (Check with ping6 on the gateway’s IPv6 address.)
Oh yes, I’m very much a command line kind of person. I use a Mac, and with all its beautiful GUI and user-friendliness, one of my favourite program is Terminal.
In case you’re a little lost about which is your public IPv6 address… it’s not the one that begins with fe80. IPv6 fe80::/10 addresses are reserved for link-local unicast addressing, which means they are only used on the directly connected network, and never routed out.
Let’s fast forward. You’ve figured out that your immediate network appears to be working alright. There are other aspects that need to be tested too, such as DNS name resolution. For now, let’s put it aside. Your computer’s network is fine. But you can’t seem to hit some website out there. You ping the faraway website, and don’t get a response. You could ping by hostname, but I often like to resolve the name as a separate step and ping by IP (or IPv6) address. So to test Facebook, I might do:
$ ping6 2a03:2880:f00c:900:face:b00c::1
That didn’t work. Is Facebook down? Well, you could try other well known servers. I tested Google’s public DNS 2001:4860:4860::8888 (the IPv6 equivalent of 8.8.8.8), and that was also not reachable. Let’s try to trace network path out to see how far we get.
$ traceroute6 2a03:2880:f00c:900:face:b00c::1
I got only as far as three hops out (first hop is the router in my network, so just two hops out into the service provider’s network). Is the service provider down? IPv4 works perfectly though.
So here we get into the interesting bit of testing reachability from elsewhere on the Internet. If you have another server elsewhere you could use, like at work or school, or your web hosting server, you can try testing from there. A traceroute back to my own IPv6 address couldn’t get very far either. By then, I can start to speculate about what might be wrong.
Let’s talk a little bit about how the Internet works. The Internet is a humongous network of networks of networks of networks, many times over. It’s like a deep hierarchy. However, many of those networks often also interconnect directly not in that hierarchical structure. In fact, networks also interconnect to each other multiple times over. It’s a really big mess.
For networks to figure out how to send traffic to each other, they use routing tables. Routers route traffic into and out of networks based on information in the routing tables. Your computer also has a routing table, although it is likely to be very small. The Internet routers in the big service providers would have route tables with some 500K route entries.
How do you determine if the route tables are alright, and that the Internet knows how to get traffic to your computer? Well, it turns out that folks on the Internet have been very helpful. There are BGP Looking Glass servers and route servers. They let you take a peek into their network to see what they see of the Internet, and what they see of your own network.
So let’s say I want to know how my network is seen by the rest of the world. First, find the public IP address of my PC. On IPv6, it is likely that you already have a public IPv6 address. In IPv4 world, make sure you check the WAN IP of your broadband router, because the LAN IP of your computer (assuming you use a broadband router) is almost surely a private IPv4 address.
Then, just go to a route server, such as route-server.he.net.
$ telnet route-server.he.net ... route-server> show ipv6 bgp 2406:3003:201b::
See if you find any routes.
When I was troubleshooting my StarHub IPv6 access issue earlier this week, there were no routes for my IPv6 prefix. If CLI is not your cup of tea, you could also use the BGP Looking Glass web interface at Hurricane Electric (choose BGP Route command), Cogent Communications (choose BGP test), or numerous other Internet carriers.
This is quite an authoritative test result for network reachability. You are looking at the actual routing tables at the very big Internet carriers. These are not ordinarily Internet Service Providers which connect to end-users. These are carriers which provide Internet backbone access to large Internet Service Providers.
Some of these carriers, such as Hurricane Electric, also provide many other tools to help to check the state of the Internet. For example, the BGP Toolkit gives you lots of information about ASes (I’ll explain later), the routes originated/announced by them, statistics, and historical information. For your interest, StarHub is AS55430, go ahead and type 55430 in the search box at the top, to find out information on StarHub’s network. SingTel is AS3578
I introduced the acronym BGP. It stands for Border Gateway Protocol. BGP is one of the many routing protocols used in networking. There are two primary classes of routing protocols: interior gateway protocols and exterior gateway protocols, with one type typically used within an organisation and the other used between organisations. BGP is what the Internet uses for exchanging routing information between Autonomous Systems (AS). Autonomous Systems typically correspond to a service provider (or group of service providers), although often very large businesses may themselves be an AS on the Internet.
I keep talking about IPv6 here, because that’s what I was troubleshooting earlier this week. The same techniques and tools apply with IPv4 (i.e. the ordinarily IP that you’re familiar with).
Have fun trying out these testing tips.